[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:40:59 dist-test-slave-03cd krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:40:59 dist-test-slave-03cd krb5kdc[399](info): set up 2 sockets
Jun 19 16:40:59 dist-test-slave-03cd krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 16:41:00.692587 392 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.006s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:41:00 dist-test-slave-03cd krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:41:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:41:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:41:00.837908 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35629
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35629
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:00.969348 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:00.969715 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:00.969801 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:00.974957 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:41:00.975068 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:00.975116 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:00.975178 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:41:00.975226 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:41:00.981730 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:00.983093 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:00.984391 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:00.995457 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:00.995509 412 server_base.cc:1034] running on GCE node
W20260619 16:41:00.995430 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:00.995437 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:00.996515 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:00.997788 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:00.998956 412 hybrid_clock.cc:648] HybridClock initialized: now 1781887260998881 us; error 91 us; skew 500 ppm
Jun 19 16:41:00 dist-test-slave-03cd krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:01.002041 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:41:01.003180 412 webserver.cc:460] Webserver started at http://127.0.98.62:40149/ using document root <none> and password file <none>
I20260619 16:41:01.003916 412 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:01.004019 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:01.004284 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:01.006595 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "e3ac7411ac2c4716914d69efd19fd55a"
format_stamp: "Formatted at 2026-06-19 16:41:01 on dist-test-slave-03cd"
server_key: "7251bcd37c7c3f8ab5a485cea735aaa9"
server_key_iv: "dfb526a7d08bec88657c05f9693d8dcc"
server_key_version: "encryptionkey@0"
I20260619 16:41:01.007300 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "e3ac7411ac2c4716914d69efd19fd55a"
format_stamp: "Formatted at 2026-06-19 16:41:01 on dist-test-slave-03cd"
server_key: "7251bcd37c7c3f8ab5a485cea735aaa9"
server_key_iv: "dfb526a7d08bec88657c05f9693d8dcc"
server_key_version: "encryptionkey@0"
I20260619 16:41:01.011587 412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:41:01.014618 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:01.016078 412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:41:01.016271 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "e3ac7411ac2c4716914d69efd19fd55a"
format_stamp: "Formatted at 2026-06-19 16:41:01 on dist-test-slave-03cd"
server_key: "7251bcd37c7c3f8ab5a485cea735aaa9"
server_key_iv: "dfb526a7d08bec88657c05f9693d8dcc"
server_key_version: "encryptionkey@0"
I20260619 16:41:01.016414 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:01.033038 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:01.037843 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:01.038120 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:01.046427 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35629
I20260619 16:41:01.046424 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35629 every 8 connection(s)
I20260619 16:41:01.048003 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:41:01.051430 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:01.056529 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 412
I20260619 16:41:01.056687 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:41:01.058104 392 external_mini_cluster.cc:1428] Setting key 587b96f9565615a09f8eafe48d1f8083
I20260619 16:41:01.058575 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3ac7411ac2c4716914d69efd19fd55a: Bootstrap starting.
I20260619 16:41:01.062144 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3ac7411ac2c4716914d69efd19fd55a: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:01.063159 480 log.cc:824] T 00000000000000000000000000000000 P e3ac7411ac2c4716914d69efd19fd55a: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:01.065407 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3ac7411ac2c4716914d69efd19fd55a: No bootstrap required, opened a new log
I20260619 16:41:01.069722 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3ac7411ac2c4716914d69efd19fd55a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ac7411ac2c4716914d69efd19fd55a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35629 } }
I20260619 16:41:01.070067 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3ac7411ac2c4716914d69efd19fd55a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:01.070147 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3ac7411ac2c4716914d69efd19fd55a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3ac7411ac2c4716914d69efd19fd55a, State: Initialized, Role: FOLLOWER
I20260619 16:41:01.070708 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3ac7411ac2c4716914d69efd19fd55a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ac7411ac2c4716914d69efd19fd55a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35629 } }
I20260619 16:41:01.070887 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3ac7411ac2c4716914d69efd19fd55a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:41:01.070968 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3ac7411ac2c4716914d69efd19fd55a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:41:01.071105 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3ac7411ac2c4716914d69efd19fd55a [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:01.072295 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3ac7411ac2c4716914d69efd19fd55a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ac7411ac2c4716914d69efd19fd55a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35629 } }
I20260619 16:41:01.072707 480 leader_election.cc:302] T 00000000000000000000000000000000 P e3ac7411ac2c4716914d69efd19fd55a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3ac7411ac2c4716914d69efd19fd55a; no voters:
I20260619 16:41:01.073091 480 leader_election.cc:288] T 00000000000000000000000000000000 P e3ac7411ac2c4716914d69efd19fd55a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:41:01.073210 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3ac7411ac2c4716914d69efd19fd55a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:01.073453 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3ac7411ac2c4716914d69efd19fd55a [term 1 LEADER]: Becoming Leader. State: Replica: e3ac7411ac2c4716914d69efd19fd55a, State: Running, Role: LEADER
I20260619 16:41:01.073865 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3ac7411ac2c4716914d69efd19fd55a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ac7411ac2c4716914d69efd19fd55a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35629 } }
I20260619 16:41:01.074502 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3ac7411ac2c4716914d69efd19fd55a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:41:01.076030 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3ac7411ac2c4716914d69efd19fd55a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3ac7411ac2c4716914d69efd19fd55a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ac7411ac2c4716914d69efd19fd55a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35629 } } }
I20260619 16:41:01.076200 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3ac7411ac2c4716914d69efd19fd55a [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:01.076429 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3ac7411ac2c4716914d69efd19fd55a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3ac7411ac2c4716914d69efd19fd55a. Latest consensus state: current_term: 1 leader_uuid: "e3ac7411ac2c4716914d69efd19fd55a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ac7411ac2c4716914d69efd19fd55a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35629 } } }
I20260619 16:41:01.076511 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3ac7411ac2c4716914d69efd19fd55a [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:01.076992 498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:41:01.080726 498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 16:41:01 dist-test-slave-03cd krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:01.088728 498 catalog_manager.cc:1347] Generated new cluster ID: fcd30c53862c419f94f039b08c92dfb1
I20260619 16:41:01.088891 498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:41:01.093544 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:01.061691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55356 (local address 127.0.98.62:35629)
0619 16:41:01.062201 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:01.062248 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:01.063425 (+ 1177us) server_negotiation.cc:407] Connection header received
0619 16:41:01.069270 (+ 5845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:01.069299 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:01.069710 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:01.070163 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:01.073487 (+ 3324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:01.075330 (+ 1843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:01.077882 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:01.078505 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:01.078682 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:01.084333 (+ 5651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:01.084341 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:01.084357 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:01.084401 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:01.086842 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:01.087572 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:01.087587 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:01.087593 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:01.087810 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:01.088246 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:01.088250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:01.088252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:01.088880 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:01.091730 (+ 2850us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:01.091922 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 16:41:01.092484 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":159,"threads_started":1}
I20260619 16:41:01.117682 498 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:41:01.118835 498 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:41:01.132629 498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3ac7411ac2c4716914d69efd19fd55a: Generated new TSK 0
I20260619 16:41:01.133548 498 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:41:01.185650 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35629
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:01.324283 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:01.324611 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:01.324658 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:01.329062 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:01.329146 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:01.329210 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:41:01.335135 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35629
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:01.336444 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:01.337647 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:01.350879 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:01.350921 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:01.351717 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:01.351725 506 server_base.cc:1034] running on GCE node
I20260619 16:41:01.352294 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:01.353562 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:01.356925 506 hybrid_clock.cc:648] HybridClock initialized: now 1781887261356840 us; error 129 us; skew 500 ppm
Jun 19 16:41:01 dist-test-slave-03cd krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:01.360780 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:41:01.361860 506 webserver.cc:460] Webserver started at http://127.0.98.1:39561/ using document root <none> and password file <none>
I20260619 16:41:01.362497 506 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:01.362565 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:01.362807 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:01.365245 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-0/data/instance:
uuid: "c8dff60029f946a88e2c695d73e2478c"
format_stamp: "Formatted at 2026-06-19 16:41:01 on dist-test-slave-03cd"
server_key: "39b76a64ad8117e79789fa3262e3ae20"
server_key_iv: "ff36716275b8e2f6fa2185f1a7e6c924"
server_key_version: "encryptionkey@0"
I20260619 16:41:01.365926 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c8dff60029f946a88e2c695d73e2478c"
format_stamp: "Formatted at 2026-06-19 16:41:01 on dist-test-slave-03cd"
server_key: "39b76a64ad8117e79789fa3262e3ae20"
server_key_iv: "ff36716275b8e2f6fa2185f1a7e6c924"
server_key_version: "encryptionkey@0"
I20260619 16:41:01.371259 506 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260619 16:41:01.375928 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:01.377619 506 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260619 16:41:01.377902 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "c8dff60029f946a88e2c695d73e2478c"
format_stamp: "Formatted at 2026-06-19 16:41:01 on dist-test-slave-03cd"
server_key: "39b76a64ad8117e79789fa3262e3ae20"
server_key_iv: "ff36716275b8e2f6fa2185f1a7e6c924"
server_key_version: "encryptionkey@0"
I20260619 16:41:01.378084 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:01.401161 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:01.404944 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:01.405187 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:01.405830 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:01.406740 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:01.406819 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:01.406889 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:01.406936 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:01.416985 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34899
I20260619 16:41:01.417011 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34899 every 8 connection(s)
I20260619 16:41:01.417935 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:41:01.424475 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 506
I20260619 16:41:01.424609 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:41:01.424877 392 external_mini_cluster.cc:1428] Setting key 139d404e87ab3dcdbda3d01848c9840a
Jun 19 16:41:01 dist-test-slave-03cd krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:01.434854 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:01.420034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58225 (local address 127.0.98.62:35629)
0619 16:41:01.420203 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:01.420207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:01.421255 (+ 1048us) server_negotiation.cc:407] Connection header received
0619 16:41:01.422227 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:01.422230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:01.422284 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:01.422368 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:01.423765 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:01.424323 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:01.426032 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:01.426369 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:01.426487 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:01.429955 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:01.429963 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:01.429970 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:01.430011 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:01.432199 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:01.432923 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:01.432929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:01.432933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:01.433081 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:01.433513 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:01.433518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:01.433520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:01.433792 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:01.433910 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:01.434484 (+ 574us) server_negotiation.cc:299] Negotiation successful
0619 16:41:01.434638 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 16:41:01.435781 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:01.420485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35629 (local address 127.0.98.1:58225)
0619 16:41:01.421013 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:01.421137 (+ 124us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:01.421991 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:01.422638 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:01.422646 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:01.423030 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:01.423546 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:01.423561 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:01.424472 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:01.424476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:01.425885 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:01.425896 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:01.426598 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:01.426603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:01.426822 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:01.427518 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:01.427543 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:01.429695 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:01.432411 (+ 2716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:01.432417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:01.432436 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:01.432774 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:01.433232 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:01.433235 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:01.433236 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:01.433398 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:01.433931 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:01.433937 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:01.434214 (+ 277us) client_negotiation.cc:769] Sending connection context
0619 16:41:01.434386 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 16:41:01.434689 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":177,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:41:01.437568 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35629
I20260619 16:41:01.437865 635 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:01.438460 635 heartbeater.cc:511] Master 127.0.98.62:35629 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 16:41:01.440264 440 ts_manager.cc:194] Registered new tserver with Master: c8dff60029f946a88e2c695d73e2478c (127.0.98.1:34899)
I20260619 16:41:01.442505 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58225
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:41:01.460798 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35629
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:01.587634 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:01.588157 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:01.588220 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:01.593793 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:01.593883 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:01.593952 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:41:01.600158 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35629
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:01.601610 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:01.602849 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:01.615397 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:01.615502 640 server_base.cc:1034] running on GCE node
W20260619 16:41:01.615388 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:01.615394 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:01.616431 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:01.617278 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:01.618573 640 hybrid_clock.cc:648] HybridClock initialized: now 1781887261618517 us; error 66 us; skew 500 ppm
Jun 19 16:41:01 dist-test-slave-03cd krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:01.622502 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:41:01.623570 640 webserver.cc:460] Webserver started at http://127.0.98.2:46563/ using document root <none> and password file <none>
I20260619 16:41:01.624229 640 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:01.624334 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:01.624615 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:01.626876 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-1/data/instance:
uuid: "a7e9db4ecaa14793801f0b79c6c1b0ce"
format_stamp: "Formatted at 2026-06-19 16:41:01 on dist-test-slave-03cd"
server_key: "c7231f32adf962c9e80ded2573499c05"
server_key_iv: "0c821632d87b5ab2cb92b1bd954ababa"
server_key_version: "encryptionkey@0"
I20260619 16:41:01.627512 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a7e9db4ecaa14793801f0b79c6c1b0ce"
format_stamp: "Formatted at 2026-06-19 16:41:01 on dist-test-slave-03cd"
server_key: "c7231f32adf962c9e80ded2573499c05"
server_key_iv: "0c821632d87b5ab2cb92b1bd954ababa"
server_key_version: "encryptionkey@0"
I20260619 16:41:01.631639 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 16:41:01.635403 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:01.637368 640 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260619 16:41:01.637583 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "a7e9db4ecaa14793801f0b79c6c1b0ce"
format_stamp: "Formatted at 2026-06-19 16:41:01 on dist-test-slave-03cd"
server_key: "c7231f32adf962c9e80ded2573499c05"
server_key_iv: "0c821632d87b5ab2cb92b1bd954ababa"
server_key_version: "encryptionkey@0"
I20260619 16:41:01.637765 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:01.670027 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:01.674026 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:01.674322 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:01.674984 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:01.676308 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:01.676391 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:01.676472 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:01.676525 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:01.689508 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41589
I20260619 16:41:01.689620 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41589 every 8 connection(s)
I20260619 16:41:01.690763 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:41:01.699277 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 640
I20260619 16:41:01.699406 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:41:01.699757 392 external_mini_cluster.cc:1428] Setting key ed09351887d348e3c227c70f5963b62f
Jun 19 16:41:01 dist-test-slave-03cd krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:01.710952 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:01.693062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48405 (local address 127.0.98.62:35629)
0619 16:41:01.693231 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:01.693236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:01.694160 (+ 924us) server_negotiation.cc:407] Connection header received
0619 16:41:01.695898 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:01.695901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:01.695956 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:01.696072 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:01.697851 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:01.698820 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:01.700765 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:01.701069 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:01.701348 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:01.705700 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:01.705705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:01.705707 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:01.705760 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:01.707491 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:01.708740 (+ 1249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:01.708744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:01.708745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:01.708901 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:01.709449 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:01.709452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:01.709453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:01.709670 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:01.709767 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:01.710684 (+ 917us) server_negotiation.cc:299] Negotiation successful
0619 16:41:01.710812 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:41:01.712283 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:01.693399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35629 (local address 127.0.98.2:48405)
0619 16:41:01.693959 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:01.694017 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:01.695523 (+ 1506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:01.696298 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:01.696368 (+ 70us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:01.696888 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:01.697579 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:01.697601 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:01.699099 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:01.699103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:01.700598 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:01.700611 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:01.701449 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:01.701456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:01.701706 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:01.702625 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:01.702655 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:01.705392 (+ 2737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:01.707793 (+ 2401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:01.707801 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:01.707828 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:01.708514 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:01.709023 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:01.709031 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:01.709035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:01.709286 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:01.709807 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:01.709818 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:01.710258 (+ 440us) client_negotiation.cc:769] Sending connection context
0619 16:41:01.710530 (+ 272us) client_negotiation.cc:240] Negotiation successful
0619 16:41:01.710987 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":118,"threads_started":1}
I20260619 16:41:01.714394 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35629
I20260619 16:41:01.714730 769 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:01.715328 769 heartbeater.cc:511] Master 127.0.98.62:35629 requested a full tablet report, sending...
I20260619 16:41:01.716593 440 ts_manager.cc:194] Registered new tserver with Master: a7e9db4ecaa14793801f0b79c6c1b0ce (127.0.98.2:41589)
I20260619 16:41:01.717260 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260619 16:41:01.963827 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.262s user 0.000s sys 0.004s
I20260619 16:41:01.964501 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35629
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:02.096349 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:02.096660 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:02.096709 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:02.101071 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:02.101147 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:02.101249 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:41:02.107097 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35629
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:02.108418 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:02.109524 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:02.120110 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:02.120110 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:02.120110 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:02.120229 774 server_base.cc:1034] running on GCE node
I20260619 16:41:02.120951 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:02.121692 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:02.122864 774 hybrid_clock.cc:648] HybridClock initialized: now 1781887262122850 us; error 27 us; skew 500 ppm
Jun 19 16:41:02 dist-test-slave-03cd krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:02.126013 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:41:02.127032 774 webserver.cc:460] Webserver started at http://127.0.98.3:43363/ using document root <none> and password file <none>
I20260619 16:41:02.127624 774 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:02.127756 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:02.127996 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:02.130229 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-2/data/instance:
uuid: "b6696e05ab654ec686d1d89252f3eabc"
format_stamp: "Formatted at 2026-06-19 16:41:02 on dist-test-slave-03cd"
server_key: "3e37abe9bcdd05aed4965e06913df2ea"
server_key_iv: "2bcad6a5bfda5dec76bf8fb8b21cf0fb"
server_key_version: "encryptionkey@0"
I20260619 16:41:02.130851 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b6696e05ab654ec686d1d89252f3eabc"
format_stamp: "Formatted at 2026-06-19 16:41:02 on dist-test-slave-03cd"
server_key: "3e37abe9bcdd05aed4965e06913df2ea"
server_key_iv: "2bcad6a5bfda5dec76bf8fb8b21cf0fb"
server_key_version: "encryptionkey@0"
I20260619 16:41:02.135156 774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:41:02.137951 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:02.139572 774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:41:02.139909 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "b6696e05ab654ec686d1d89252f3eabc"
format_stamp: "Formatted at 2026-06-19 16:41:02 on dist-test-slave-03cd"
server_key: "3e37abe9bcdd05aed4965e06913df2ea"
server_key_iv: "2bcad6a5bfda5dec76bf8fb8b21cf0fb"
server_key_version: "encryptionkey@0"
I20260619 16:41:02.140065 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:02.172407 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:02.176234 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:02.176465 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:02.177100 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:02.177986 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:02.178062 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:02.178133 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:02.178180 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:02.189697 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44101
I20260619 16:41:02.189713 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44101 every 8 connection(s)
I20260619 16:41:02.190855 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:41:02.192627 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 774
I20260619 16:41:02.192785 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887259574980-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:41:02.193135 392 external_mini_cluster.cc:1428] Setting key 141d81c396f72f84febc742cbb17d8c0
Jun 19 16:41:02 dist-test-slave-03cd krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:02.206805 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:02.193702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49905 (local address 127.0.98.62:35629)
0619 16:41:02.193876 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:02.193882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:02.194768 (+ 886us) server_negotiation.cc:407] Connection header received
0619 16:41:02.196261 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:02.196264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:02.196315 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:02.196389 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:02.197673 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:02.198197 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:02.199440 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:02.199783 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:02.199886 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:02.202667 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:02.202671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:02.202673 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:02.202712 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:02.204438 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:02.205107 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:02.205111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:02.205113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:02.205216 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:02.205605 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:02.205608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:02.205609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:02.205814 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:02.205903 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:02.206499 (+ 596us) server_negotiation.cc:299] Negotiation successful
0619 16:41:02.206623 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 16:41:02.208024 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:02.194075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35629 (local address 127.0.98.3:49905)
0619 16:41:02.194616 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:02.194674 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:02.196081 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:02.196560 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:02.196568 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:02.196972 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:02.197471 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:02.197486 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:02.198330 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:02.198333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:02.199267 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:02.199277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:02.199974 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:02.199978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:02.200143 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:02.200769 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:02.200787 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:02.202465 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:02.204599 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:02.204605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:02.204622 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:02.204929 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:02.205320 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:02.205322 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:02.205323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:02.205506 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:02.205919 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:02.205924 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:02.206195 (+ 271us) client_negotiation.cc:769] Sending connection context
0619 16:41:02.206363 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 16:41:02.206797 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":158,"threads_started":1}
I20260619 16:41:02.209218 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35629
I20260619 16:41:02.209548 903 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:02.210127 903 heartbeater.cc:511] Master 127.0.98.62:35629 requested a full tablet report, sending...
I20260619 16:41:02.211323 440 ts_manager.cc:194] Registered new tserver with Master: b6696e05ab654ec686d1d89252f3eabc (127.0.98.3:44101)
I20260619 16:41:02.212018 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49905
I20260619 16:41:02.220847 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:41:02 dist-test-slave-03cd krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887262, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:41:02 dist-test-slave-03cd krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887262, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:41:02.249940 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:02.239060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40458 (local address 127.0.98.1:34899)
0619 16:41:02.239407 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:02.239413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:02.239498 (+ 85us) server_negotiation.cc:407] Connection header received
0619 16:41:02.239607 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:02.239613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:02.239866 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:02.239990 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:02.240684 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:02.241451 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:02.242389 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:02.242826 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:02.242916 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:02.246052 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:02.246063 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:02.246082 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:02.246113 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:02.248103 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:02.248560 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:02.248564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:02.248566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:02.248652 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:02.248968 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:02.248970 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:02.248971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:02.249227 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:02.249402 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:02.249637 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 16:41:02.249772 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":139,"threads_started":1}
W20260619 16:41:02.250941 546 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40458
Jun 19 16:41:02 dist-test-slave-03cd krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:41:02 dist-test-slave-03cd krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:41:02.278734 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:02.269110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40464 (local address 127.0.98.1:34899)
0619 16:41:02.269323 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:02.269327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:02.269403 (+ 76us) server_negotiation.cc:407] Connection header received
0619 16:41:02.269539 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:02.269542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:02.269583 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:02.269666 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:02.270526 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:02.271071 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:02.272087 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:02.272317 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:02.272405 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:02.275415 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:02.275419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:02.275421 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:02.275483 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:02.277062 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:02.277522 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:02.277525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:02.277526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:02.277613 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:02.277913 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:02.277915 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:02.277916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:02.278090 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:02.278190 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:02.278412 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 16:41:02.278525 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260619 16:41:02.280047 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 506
I20260619 16:41:02.286079 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 640
I20260619 16:41:02.294081 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 774
I20260619 16:41:02.299645 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 412
2026-06-19T16:41:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2712 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:41:02 dist-test-slave-03cd krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:41:02 dist-test-slave-03cd krb5kdc[917](info): set up 2 sockets
Jun 19 16:41:02 dist-test-slave-03cd krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260619 16:41:03.388172 392 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:41:03 dist-test-slave-03cd krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:41:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:41:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:41:03.528870 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42209
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:03.650172 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:03.650564 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:03.650648 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:03.655337 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:41:03.655455 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:03.655490 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:03.655529 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:41:03.655565 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:41:03.663271 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:03.664605 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:03.665889 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:03.675421 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:03.675421 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:03.675475 930 server_base.cc:1034] running on GCE node
W20260619 16:41:03.675426 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:03.676774 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:03.678148 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:03.679422 930 hybrid_clock.cc:648] HybridClock initialized: now 1781887263679388 us; error 52 us; skew 500 ppm
Jun 19 16:41:03 dist-test-slave-03cd krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:03.683020 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:41:03.684079 930 webserver.cc:460] Webserver started at http://127.0.98.62:46131/ using document root <none> and password file <none>
I20260619 16:41:03.684640 930 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:03.684731 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:03.684906 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:03.687033 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "80897b80bd4e48cf9a0b60d154dc2dc5"
format_stamp: "Formatted at 2026-06-19 16:41:03 on dist-test-slave-03cd"
server_key: "a2e9d1c8d021d61230d1ef5de3a833ad"
server_key_iv: "6502027b68bb5c65b9cbca3a506ec274"
server_key_version: "encryptionkey@0"
I20260619 16:41:03.687595 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "80897b80bd4e48cf9a0b60d154dc2dc5"
format_stamp: "Formatted at 2026-06-19 16:41:03 on dist-test-slave-03cd"
server_key: "a2e9d1c8d021d61230d1ef5de3a833ad"
server_key_iv: "6502027b68bb5c65b9cbca3a506ec274"
server_key_version: "encryptionkey@0"
I20260619 16:41:03.691872 930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:41:03.694540 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:03.696113 930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:41:03.696271 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "80897b80bd4e48cf9a0b60d154dc2dc5"
format_stamp: "Formatted at 2026-06-19 16:41:03 on dist-test-slave-03cd"
server_key: "a2e9d1c8d021d61230d1ef5de3a833ad"
server_key_iv: "6502027b68bb5c65b9cbca3a506ec274"
server_key_version: "encryptionkey@0"
I20260619 16:41:03.696403 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:03.706907 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:03.710541 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:03.710772 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:03.718055 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42209 every 8 connection(s)
I20260619 16:41:03.718055 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42209
I20260619 16:41:03.719245 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:41:03.722419 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:03.726512 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 930
I20260619 16:41:03.726642 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:41:03.726910 392 external_mini_cluster.cc:1428] Setting key 88c3fbe2fa0bfc381afbc577c9821987
I20260619 16:41:03.729111 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80897b80bd4e48cf9a0b60d154dc2dc5: Bootstrap starting.
I20260619 16:41:03.731779 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80897b80bd4e48cf9a0b60d154dc2dc5: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:03.732524 998 log.cc:824] T 00000000000000000000000000000000 P 80897b80bd4e48cf9a0b60d154dc2dc5: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:03.734803 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80897b80bd4e48cf9a0b60d154dc2dc5: No bootstrap required, opened a new log
Jun 19 16:41:03 dist-test-slave-03cd krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:03.739625 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80897b80bd4e48cf9a0b60d154dc2dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80897b80bd4e48cf9a0b60d154dc2dc5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42209 } }
I20260619 16:41:03.740000 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80897b80bd4e48cf9a0b60d154dc2dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:03.740085 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80897b80bd4e48cf9a0b60d154dc2dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80897b80bd4e48cf9a0b60d154dc2dc5, State: Initialized, Role: FOLLOWER
I20260619 16:41:03.740561 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80897b80bd4e48cf9a0b60d154dc2dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80897b80bd4e48cf9a0b60d154dc2dc5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42209 } }
I20260619 16:41:03.740729 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80897b80bd4e48cf9a0b60d154dc2dc5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:41:03.740808 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80897b80bd4e48cf9a0b60d154dc2dc5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:41:03.740916 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80897b80bd4e48cf9a0b60d154dc2dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:03.741906 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80897b80bd4e48cf9a0b60d154dc2dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80897b80bd4e48cf9a0b60d154dc2dc5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42209 } }
I20260619 16:41:03.742305 998 leader_election.cc:302] T 00000000000000000000000000000000 P 80897b80bd4e48cf9a0b60d154dc2dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80897b80bd4e48cf9a0b60d154dc2dc5; no voters:
I20260619 16:41:03.742673 998 leader_election.cc:288] T 00000000000000000000000000000000 P 80897b80bd4e48cf9a0b60d154dc2dc5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:41:03.742748 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80897b80bd4e48cf9a0b60d154dc2dc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:03.742957 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80897b80bd4e48cf9a0b60d154dc2dc5 [term 1 LEADER]: Becoming Leader. State: Replica: 80897b80bd4e48cf9a0b60d154dc2dc5, State: Running, Role: LEADER
I20260619 16:41:03.743218 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:03.728539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33836 (local address 127.0.98.62:42209)
0619 16:41:03.729023 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:03.729039 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:03.729074 (+ 35us) server_negotiation.cc:407] Connection header received
0619 16:41:03.729780 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:03.729799 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:03.730143 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:03.730438 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:03.731402 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:03.732596 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:03.733748 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:03.734257 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:03.734387 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:03.737424 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:03.737431 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:03.737446 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:03.737475 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:03.739531 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:03.740209 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:03.740216 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:03.740223 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:03.740398 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:03.740729 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:03.740732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:03.740734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:03.741280 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:03.741518 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:03.741810 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 16:41:03.742157 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":173,"threads_started":1}
I20260619 16:41:03.743283 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80897b80bd4e48cf9a0b60d154dc2dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80897b80bd4e48cf9a0b60d154dc2dc5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42209 } }
I20260619 16:41:03.744076 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80897b80bd4e48cf9a0b60d154dc2dc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:41:03.746050 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80897b80bd4e48cf9a0b60d154dc2dc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80897b80bd4e48cf9a0b60d154dc2dc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80897b80bd4e48cf9a0b60d154dc2dc5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42209 } } }
I20260619 16:41:03.746191 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80897b80bd4e48cf9a0b60d154dc2dc5 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:03.746546 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80897b80bd4e48cf9a0b60d154dc2dc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80897b80bd4e48cf9a0b60d154dc2dc5. Latest consensus state: current_term: 1 leader_uuid: "80897b80bd4e48cf9a0b60d154dc2dc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80897b80bd4e48cf9a0b60d154dc2dc5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42209 } } }
I20260619 16:41:03.746636 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80897b80bd4e48cf9a0b60d154dc2dc5 [sys.catalog]: This master's current role is: LEADER
W20260619 16:41:03.750221 1019 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 80897b80bd4e48cf9a0b60d154dc2dc5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:41:03.750356 1019 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:41:03.750428 1013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:41:03.751212 1013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:41:03.757689 1013 catalog_manager.cc:1347] Generated new cluster ID: 58238a0d5c044c149e91deaf019f6e8e
I20260619 16:41:03.757793 1013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:41:03.791090 1013 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:41:03.792536 1013 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:41:03.806175 1013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80897b80bd4e48cf9a0b60d154dc2dc5: Generated new TSK 0
I20260619 16:41:03.806970 1013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:41:03.840543 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42209
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:03.964234 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:03.964540 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:03.964586 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:03.969967 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:03.970077 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:03.970145 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:41:03.977591 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42209
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:03.979094 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:03.980495 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:03.992255 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:03.992237 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:03.992309 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:03.993468 1024 server_base.cc:1034] running on GCE node
I20260619 16:41:03.994014 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:03.994765 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:03.996012 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781887263995994 us; error 33 us; skew 500 ppm
Jun 19 16:41:03 dist-test-slave-03cd krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:04.000036 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:41:04.001686 1024 webserver.cc:460] Webserver started at http://127.0.98.1:46499/ using document root <none> and password file <none>
I20260619 16:41:04.002378 1024 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:04.002492 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:04.002744 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:04.005059 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-0/data/instance:
uuid: "204eff21de4d47f1aa3163d96207c97c"
format_stamp: "Formatted at 2026-06-19 16:41:03 on dist-test-slave-03cd"
server_key: "c8f3c8215230e2742a2f2f010066dae9"
server_key_iv: "aaed50cf604a6f0b4a856c39ac606ac9"
server_key_version: "encryptionkey@0"
I20260619 16:41:04.005966 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-0/wal/instance:
uuid: "204eff21de4d47f1aa3163d96207c97c"
format_stamp: "Formatted at 2026-06-19 16:41:03 on dist-test-slave-03cd"
server_key: "c8f3c8215230e2742a2f2f010066dae9"
server_key_iv: "aaed50cf604a6f0b4a856c39ac606ac9"
server_key_version: "encryptionkey@0"
I20260619 16:41:04.011368 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:41:04.015069 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:04.016645 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:41:04.016851 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "204eff21de4d47f1aa3163d96207c97c"
format_stamp: "Formatted at 2026-06-19 16:41:03 on dist-test-slave-03cd"
server_key: "c8f3c8215230e2742a2f2f010066dae9"
server_key_iv: "aaed50cf604a6f0b4a856c39ac606ac9"
server_key_version: "encryptionkey@0"
I20260619 16:41:04.017005 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:04.033752 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:04.037487 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:04.037724 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:04.038373 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:04.039248 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:04.039328 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:04.039429 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:04.039484 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:04.050274 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36429
I20260619 16:41:04.050323 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36429 every 8 connection(s)
I20260619 16:41:04.051712 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:41:04.058780 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 1024
I20260619 16:41:04.058941 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:41:04.059204 392 external_mini_cluster.cc:1428] Setting key e2d9e20b781ac85e0005052b2a4cf0c3
Jun 19 16:41:04 dist-test-slave-03cd krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:41:04.069977 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:04.054592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51185 (local address 127.0.98.62:42209)
0619 16:41:04.054747 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:04.054756 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:04.055696 (+ 940us) server_negotiation.cc:407] Connection header received
0619 16:41:04.057150 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:04.057153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:04.057206 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:04.057292 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:04.058856 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.059522 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.061112 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.061353 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.061480 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:04.065299 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:04.065303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:04.065305 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:04.065338 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:04.067041 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:04.067918 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:04.067921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:04.067923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:04.068020 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:04.068511 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:04.068514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:04.068515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:04.068724 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:04.068804 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:04.069691 (+ 887us) server_negotiation.cc:299] Negotiation successful
0619 16:41:04.069830 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:41:04.071110 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:04.054862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42209 (local address 127.0.98.1:51185)
0619 16:41:04.055453 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:04.055512 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:04.056886 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:04.057579 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:04.057587 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:04.058030 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:04.058600 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:04.058615 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.059733 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.059737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:04.060977 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:04.060989 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.061530 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.061536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:04.061725 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:04.062730 (+ 1005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:04.062795 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:04.065043 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:04.067213 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:04.067221 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:04.067244 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:04.067709 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:04.068146 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:04.068152 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:04.068157 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:04.068382 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:04.068860 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:04.068871 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:04.069259 (+ 388us) client_negotiation.cc:769] Sending connection context
0619 16:41:04.069510 (+ 251us) client_negotiation.cc:240] Negotiation successful
0619 16:41:04.069860 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":141,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 16:41:04.073432 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42209
I20260619 16:41:04.073935 1153 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:04.074604 1153 heartbeater.cc:511] Master 127.0.98.62:42209 requested a full tablet report, sending...
I20260619 16:41:04.076390 962 ts_manager.cc:194] Registered new tserver with Master: 204eff21de4d47f1aa3163d96207c97c (127.0.98.1:36429)
I20260619 16:41:04.077688 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51185
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:41:04.092433 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42209
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:04.210737 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:04.211102 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:04.211185 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:04.215534 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:04.215672 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:04.215801 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:41:04.221616 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42209
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:04.222875 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:04.223956 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:04.234637 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:04.234674 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:04.234674 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:04.234865 1158 server_base.cc:1034] running on GCE node
I20260619 16:41:04.235285 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:04.236045 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:04.237223 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781887264237215 us; error 30 us; skew 500 ppm
Jun 19 16:41:04 dist-test-slave-03cd krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:04.240491 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:41:04.241556 1158 webserver.cc:460] Webserver started at http://127.0.98.2:37067/ using document root <none> and password file <none>
I20260619 16:41:04.242302 1158 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:04.242460 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:04.242743 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:04.245257 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-1/data/instance:
uuid: "bac96cb958244e309e798b2be0ef4572"
format_stamp: "Formatted at 2026-06-19 16:41:04 on dist-test-slave-03cd"
server_key: "8788d02c127cc884b8700951f63a18e7"
server_key_iv: "a3b3b2a3a69eddf55b11b8b26ea54203"
server_key_version: "encryptionkey@0"
I20260619 16:41:04.245931 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bac96cb958244e309e798b2be0ef4572"
format_stamp: "Formatted at 2026-06-19 16:41:04 on dist-test-slave-03cd"
server_key: "8788d02c127cc884b8700951f63a18e7"
server_key_iv: "a3b3b2a3a69eddf55b11b8b26ea54203"
server_key_version: "encryptionkey@0"
I20260619 16:41:04.250277 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:41:04.253047 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:04.254400 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:41:04.254571 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "bac96cb958244e309e798b2be0ef4572"
format_stamp: "Formatted at 2026-06-19 16:41:04 on dist-test-slave-03cd"
server_key: "8788d02c127cc884b8700951f63a18e7"
server_key_iv: "a3b3b2a3a69eddf55b11b8b26ea54203"
server_key_version: "encryptionkey@0"
I20260619 16:41:04.254709 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:04.272428 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:04.276253 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:04.276492 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:04.277115 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:04.277995 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:04.278074 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:04.278142 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:04.278196 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:04.288166 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38499
I20260619 16:41:04.288190 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38499 every 8 connection(s)
I20260619 16:41:04.289258 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:41:04.289844 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 1158
I20260619 16:41:04.289971 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:41:04.290274 392 external_mini_cluster.cc:1428] Setting key ada2fa063856e2ae925a237bdc1032cd
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Jun 19 16:41:04 dist-test-slave-03cd krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:04.312037 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:04.291477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50517 (local address 127.0.98.62:42209)
0619 16:41:04.291646 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:04.291651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:04.293431 (+ 1780us) server_negotiation.cc:407] Connection header received
0619 16:41:04.294901 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:04.294904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:04.294952 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:04.295031 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:04.296926 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.297462 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.299447 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.299778 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.299910 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:04.306634 (+ 6724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:04.306638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:04.306641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:04.306696 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:04.308783 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:04.309772 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:04.309777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:04.309778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:04.309886 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:04.310464 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:04.310471 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:04.310474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:04.310825 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:04.310954 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:04.311618 (+ 664us) server_negotiation.cc:299] Negotiation successful
0619 16:41:04.311832 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:41:04.312960 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:04.292023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42209 (local address 127.0.98.2:50517)
0619 16:41:04.293239 (+ 1216us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:04.293299 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:04.294597 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:04.295273 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:04.295285 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:04.295917 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:04.296721 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:04.296759 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.297626 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.297634 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:04.299270 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:04.299286 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.300011 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.300016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:04.300252 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:04.301024 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:04.301047 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:04.306374 (+ 5327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:04.309114 (+ 2740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:04.309121 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:04.309137 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:04.309598 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:04.310041 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:04.310044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:04.310053 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:04.310309 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:04.310980 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:04.310987 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:04.311320 (+ 333us) client_negotiation.cc:769] Sending connection context
0619 16:41:04.311503 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 16:41:04.311957 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"thread_start_us":758,"threads_started":1}
I20260619 16:41:04.314452 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42209
I20260619 16:41:04.314772 1287 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:04.315407 1287 heartbeater.cc:511] Master 127.0.98.62:42209 requested a full tablet report, sending...
I20260619 16:41:04.316814 962 ts_manager.cc:194] Registered new tserver with Master: bac96cb958244e309e798b2be0ef4572 (127.0.98.2:38499)
I20260619 16:41:04.317457 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:41:04.323309 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42209
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:04.446909 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:04.447260 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:04.447309 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:04.451992 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:04.452085 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:04.452155 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:41:04.458039 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42209
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:04.459339 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:04.460505 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:41:04.471866 1292 server_base.cc:1034] running on GCE node
W20260619 16:41:04.471807 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:04.471820 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:04.471807 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:04.472710 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:04.473387 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:04.474617 1292 hybrid_clock.cc:648] HybridClock initialized: now 1781887264474604 us; error 29 us; skew 500 ppm
Jun 19 16:41:04 dist-test-slave-03cd krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:04.482156 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:41:04.483248 1292 webserver.cc:460] Webserver started at http://127.0.98.3:43313/ using document root <none> and password file <none>
I20260619 16:41:04.483966 1292 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:04.484038 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:04.484277 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:04.486603 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-2/data/instance:
uuid: "a123c4504ef644de802def111f41624b"
format_stamp: "Formatted at 2026-06-19 16:41:04 on dist-test-slave-03cd"
server_key: "d13808b0069d9939691c2addf011297f"
server_key_iv: "5b5b754bac202fd7fcc7d6dda3a1f265"
server_key_version: "encryptionkey@0"
I20260619 16:41:04.487229 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a123c4504ef644de802def111f41624b"
format_stamp: "Formatted at 2026-06-19 16:41:04 on dist-test-slave-03cd"
server_key: "d13808b0069d9939691c2addf011297f"
server_key_iv: "5b5b754bac202fd7fcc7d6dda3a1f265"
server_key_version: "encryptionkey@0"
I20260619 16:41:04.491539 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:41:04.494689 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:04.496480 1292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:41:04.496621 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "a123c4504ef644de802def111f41624b"
format_stamp: "Formatted at 2026-06-19 16:41:04 on dist-test-slave-03cd"
server_key: "d13808b0069d9939691c2addf011297f"
server_key_iv: "5b5b754bac202fd7fcc7d6dda3a1f265"
server_key_version: "encryptionkey@0"
I20260619 16:41:04.496809 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:04.530614 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:04.534293 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:04.534478 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:04.535059 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:04.536204 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:04.536264 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:04.536309 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:04.536324 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:04.547345 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34921
I20260619 16:41:04.547504 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34921 every 8 connection(s)
I20260619 16:41:04.548513 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:41:04.551498 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 1292
I20260619 16:41:04.551690 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887259574980-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:41:04.552037 392 external_mini_cluster.cc:1428] Setting key fb12229a2cb7b313433600f7da3b0355
Jun 19 16:41:04 dist-test-slave-03cd krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:04.564481 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:04.550759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49189 (local address 127.0.98.62:42209)
0619 16:41:04.550932 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:04.550937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:04.552322 (+ 1385us) server_negotiation.cc:407] Connection header received
0619 16:41:04.554044 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:04.554047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:04.554102 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:04.554175 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:04.555444 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.556072 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.557262 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.557502 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.557628 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:04.560505 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:04.560509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:04.560512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:04.560547 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:04.562092 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:04.562708 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:04.562712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:04.562714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:04.562836 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:04.563224 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:04.563229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:04.563231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:04.563450 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:04.563565 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:04.564175 (+ 610us) server_negotiation.cc:299] Negotiation successful
0619 16:41:04.564311 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:41:04.565488 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:04.551402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42209 (local address 127.0.98.3:49189)
0619 16:41:04.552112 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:04.552182 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:04.553818 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:04.554361 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:04.554368 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:04.554768 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:04.555289 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:04.555302 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.556207 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.556212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:04.557122 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:04.557131 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.557665 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.557669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:04.557871 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:04.558503 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:04.558520 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:04.560300 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:04.562265 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:04.562272 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:04.562302 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:04.562584 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:04.562948 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:04.562951 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:04.562953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:04.563117 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:04.563574 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:04.563583 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:04.563910 (+ 327us) client_negotiation.cc:769] Sending connection context
0619 16:41:04.564118 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 16:41:04.564475 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":224,"threads_started":1}
I20260619 16:41:04.566740 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42209
I20260619 16:41:04.567001 1421 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:04.567513 1421 heartbeater.cc:511] Master 127.0.98.62:42209 requested a full tablet report, sending...
I20260619 16:41:04.568771 961 ts_manager.cc:194] Registered new tserver with Master: a123c4504ef644de802def111f41624b (127.0.98.3:34921)
I20260619 16:41:04.569394 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49189
I20260619 16:41:04.579046 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:41:04.591873 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:04.582899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56558 (local address 127.0.98.62:42209)
0619 16:41:04.583076 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:04.583079 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:04.583147 (+ 68us) server_negotiation.cc:407] Connection header received
0619 16:41:04.583355 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:04.583359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:04.583407 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:04.583486 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:04.584356 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.584900 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.585780 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.586006 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.586092 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:04.587603 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:04.587606 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:04.587608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:04.587636 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:04.589427 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:04.589943 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:04.589946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:04.589947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:04.590041 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:04.590752 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:04.590755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:04.590757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:04.591029 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:04.591149 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:04.591520 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 16:41:04.591634 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 16:41:04.598347 961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:41:04.602455 961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:41:04.618798 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:04.613099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48440 (local address 127.0.98.2:38499)
0619 16:41:04.613748 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:04.613759 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:04.613785 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:41:04.613872 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:04.613877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:04.614109 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:04.614278 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:04.614628 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.615611 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.617352 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.618036 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.618178 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:04.618347 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:04.618525 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 16:41:04.618623 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":117,"threads_started":1}
I20260619 16:41:04.618809 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:04.612359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38499 (local address 127.0.0.1:48440)
0619 16:41:04.613035 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:04.613093 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:04.613230 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:04.614297 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:04.614309 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:04.614331 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:04.614454 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:04.614489 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.615816 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.615819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:04.617206 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:04.617217 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.618190 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.618193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:04.618311 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:04.618350 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 16:41:04.618495 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 16:41:04.618627 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":102,"threads_started":1}
I20260619 16:41:04.618893 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:04.612754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34921 (local address 127.0.0.1:59992)
0619 16:41:04.613963 (+ 1209us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:04.613979 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:04.614067 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:04.614522 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:04.614528 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:04.614549 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:04.614643 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:04.614648 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.616115 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.616118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:04.617253 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:04.617263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.618443 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.618447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:04.618545 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:04.618569 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 16:41:04.618635 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 16:41:04.618716 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"mutex_wait_us":1,"thread_start_us":80,"threads_started":1}
I20260619 16:41:04.619079 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:04.612627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36429 (local address 127.0.0.1:47978)
0619 16:41:04.613634 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:04.613649 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:04.613750 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:04.614927 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:04.614930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:04.614950 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:04.615041 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:04.615046 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.616417 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.616420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:04.617522 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:04.617532 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.618706 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.618709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:04.618787 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:04.618808 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 16:41:04.618875 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 16:41:04.618949 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"mutex_wait_us":1,"thread_start_us":113,"threads_started":1}
I20260619 16:41:04.619498 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:04.613196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47978 (local address 127.0.98.1:36429)
0619 16:41:04.613549 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:04.613557 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:04.614455 (+ 898us) server_negotiation.cc:407] Connection header received
0619 16:41:04.614535 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:04.614540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:04.614783 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:04.614931 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:04.615171 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.616285 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.617659 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.618428 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.618528 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:04.618653 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:04.619250 (+ 597us) server_negotiation.cc:299] Negotiation successful
0619 16:41:04.619358 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":74,"threads_started":1}
I20260619 16:41:04.619496 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:04.613186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59992 (local address 127.0.98.3:34921)
0619 16:41:04.613979 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:04.613990 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:04.614085 (+ 95us) server_negotiation.cc:407] Connection header received
0619 16:41:04.614160 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:04.614170 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:04.614377 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:04.614522 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:04.614774 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.615976 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.617372 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.618134 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.618240 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:04.618376 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:04.619250 (+ 874us) server_negotiation.cc:299] Negotiation successful
0619 16:41:04.619347 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":158,"threads_started":1}
I20260619 16:41:04.622494 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 831458ca0817495dba8b124852e07d0a (DEFAULT_TABLE table=test-table [id=430535953767456eab5f8b1776da8655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:04.622495 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 831458ca0817495dba8b124852e07d0a (DEFAULT_TABLE table=test-table [id=430535953767456eab5f8b1776da8655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:04.622494 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 831458ca0817495dba8b124852e07d0a (DEFAULT_TABLE table=test-table [id=430535953767456eab5f8b1776da8655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:04.623633 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 831458ca0817495dba8b124852e07d0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:04.623633 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 831458ca0817495dba8b124852e07d0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:04.623628 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 831458ca0817495dba8b124852e07d0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:04.630390 1438 tablet_bootstrap.cc:492] T 831458ca0817495dba8b124852e07d0a P a123c4504ef644de802def111f41624b: Bootstrap starting.
I20260619 16:41:04.632071 1439 tablet_bootstrap.cc:492] T 831458ca0817495dba8b124852e07d0a P 204eff21de4d47f1aa3163d96207c97c: Bootstrap starting.
I20260619 16:41:04.632937 1438 tablet_bootstrap.cc:654] T 831458ca0817495dba8b124852e07d0a P a123c4504ef644de802def111f41624b: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:04.634095 1438 log.cc:824] T 831458ca0817495dba8b124852e07d0a P a123c4504ef644de802def111f41624b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:04.634624 1439 tablet_bootstrap.cc:654] T 831458ca0817495dba8b124852e07d0a P 204eff21de4d47f1aa3163d96207c97c: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:04.634932 1440 tablet_bootstrap.cc:492] T 831458ca0817495dba8b124852e07d0a P bac96cb958244e309e798b2be0ef4572: Bootstrap starting.
I20260619 16:41:04.635504 1439 log.cc:824] T 831458ca0817495dba8b124852e07d0a P 204eff21de4d47f1aa3163d96207c97c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:04.636778 1438 tablet_bootstrap.cc:492] T 831458ca0817495dba8b124852e07d0a P a123c4504ef644de802def111f41624b: No bootstrap required, opened a new log
I20260619 16:41:04.637029 1438 ts_tablet_manager.cc:1397] T 831458ca0817495dba8b124852e07d0a P a123c4504ef644de802def111f41624b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 16:41:04.637775 1440 tablet_bootstrap.cc:654] T 831458ca0817495dba8b124852e07d0a P bac96cb958244e309e798b2be0ef4572: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:04.637897 1439 tablet_bootstrap.cc:492] T 831458ca0817495dba8b124852e07d0a P 204eff21de4d47f1aa3163d96207c97c: No bootstrap required, opened a new log
I20260619 16:41:04.638093 1439 ts_tablet_manager.cc:1397] T 831458ca0817495dba8b124852e07d0a P 204eff21de4d47f1aa3163d96207c97c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 16:41:04.638655 1440 log.cc:824] T 831458ca0817495dba8b124852e07d0a P bac96cb958244e309e798b2be0ef4572: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:04.640588 1438 raft_consensus.cc:348] T 831458ca0817495dba8b124852e07d0a P a123c4504ef644de802def111f41624b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac96cb958244e309e798b2be0ef4572" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38499 } } peers { permanent_uuid: "204eff21de4d47f1aa3163d96207c97c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36429 } } peers { permanent_uuid: "a123c4504ef644de802def111f41624b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34921 } }
I20260619 16:41:04.640888 1438 raft_consensus.cc:374] T 831458ca0817495dba8b124852e07d0a P a123c4504ef644de802def111f41624b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:04.640946 1438 raft_consensus.cc:729] T 831458ca0817495dba8b124852e07d0a P a123c4504ef644de802def111f41624b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a123c4504ef644de802def111f41624b, State: Initialized, Role: FOLLOWER
I20260619 16:41:04.641129 1440 tablet_bootstrap.cc:492] T 831458ca0817495dba8b124852e07d0a P bac96cb958244e309e798b2be0ef4572: No bootstrap required, opened a new log
I20260619 16:41:04.641345 1440 ts_tablet_manager.cc:1397] T 831458ca0817495dba8b124852e07d0a P bac96cb958244e309e798b2be0ef4572: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 16:41:04.641497 1438 consensus_queue.cc:260] T 831458ca0817495dba8b124852e07d0a P a123c4504ef644de802def111f41624b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac96cb958244e309e798b2be0ef4572" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38499 } } peers { permanent_uuid: "204eff21de4d47f1aa3163d96207c97c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36429 } } peers { permanent_uuid: "a123c4504ef644de802def111f41624b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34921 } }
I20260619 16:41:04.642222 1421 heartbeater.cc:503] Master 127.0.98.62:42209 was elected leader, sending a full tablet report...
I20260619 16:41:04.643014 1438 ts_tablet_manager.cc:1428] T 831458ca0817495dba8b124852e07d0a P a123c4504ef644de802def111f41624b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 16:41:04.644024 1439 raft_consensus.cc:348] T 831458ca0817495dba8b124852e07d0a P 204eff21de4d47f1aa3163d96207c97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac96cb958244e309e798b2be0ef4572" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38499 } } peers { permanent_uuid: "204eff21de4d47f1aa3163d96207c97c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36429 } } peers { permanent_uuid: "a123c4504ef644de802def111f41624b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34921 } }
I20260619 16:41:04.644434 1439 raft_consensus.cc:374] T 831458ca0817495dba8b124852e07d0a P 204eff21de4d47f1aa3163d96207c97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:04.644495 1439 raft_consensus.cc:729] T 831458ca0817495dba8b124852e07d0a P 204eff21de4d47f1aa3163d96207c97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 204eff21de4d47f1aa3163d96207c97c, State: Initialized, Role: FOLLOWER
I20260619 16:41:04.644981 1439 consensus_queue.cc:260] T 831458ca0817495dba8b124852e07d0a P 204eff21de4d47f1aa3163d96207c97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac96cb958244e309e798b2be0ef4572" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38499 } } peers { permanent_uuid: "204eff21de4d47f1aa3163d96207c97c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36429 } } peers { permanent_uuid: "a123c4504ef644de802def111f41624b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34921 } }
I20260619 16:41:04.645571 1440 raft_consensus.cc:348] T 831458ca0817495dba8b124852e07d0a P bac96cb958244e309e798b2be0ef4572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac96cb958244e309e798b2be0ef4572" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38499 } } peers { permanent_uuid: "204eff21de4d47f1aa3163d96207c97c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36429 } } peers { permanent_uuid: "a123c4504ef644de802def111f41624b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34921 } }
I20260619 16:41:04.645850 1440 raft_consensus.cc:374] T 831458ca0817495dba8b124852e07d0a P bac96cb958244e309e798b2be0ef4572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:04.645898 1440 raft_consensus.cc:729] T 831458ca0817495dba8b124852e07d0a P bac96cb958244e309e798b2be0ef4572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bac96cb958244e309e798b2be0ef4572, State: Initialized, Role: FOLLOWER
I20260619 16:41:04.646333 1439 ts_tablet_manager.cc:1428] T 831458ca0817495dba8b124852e07d0a P 204eff21de4d47f1aa3163d96207c97c: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260619 16:41:04.646389 1440 consensus_queue.cc:260] T 831458ca0817495dba8b124852e07d0a P bac96cb958244e309e798b2be0ef4572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac96cb958244e309e798b2be0ef4572" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38499 } } peers { permanent_uuid: "204eff21de4d47f1aa3163d96207c97c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36429 } } peers { permanent_uuid: "a123c4504ef644de802def111f41624b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34921 } }
I20260619 16:41:04.646740 1153 heartbeater.cc:503] Master 127.0.98.62:42209 was elected leader, sending a full tablet report...
I20260619 16:41:04.647192 1287 heartbeater.cc:503] Master 127.0.98.62:42209 was elected leader, sending a full tablet report...
I20260619 16:41:04.647349 1440 ts_tablet_manager.cc:1428] T 831458ca0817495dba8b124852e07d0a P bac96cb958244e309e798b2be0ef4572: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 16:41:04.748137 1446 raft_consensus.cc:482] T 831458ca0817495dba8b124852e07d0a P bac96cb958244e309e798b2be0ef4572 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:41:04.748435 1446 raft_consensus.cc:504] T 831458ca0817495dba8b124852e07d0a P bac96cb958244e309e798b2be0ef4572 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac96cb958244e309e798b2be0ef4572" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38499 } } peers { permanent_uuid: "204eff21de4d47f1aa3163d96207c97c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36429 } } peers { permanent_uuid: "a123c4504ef644de802def111f41624b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34921 } }
I20260619 16:41:04.749794 1446 leader_election.cc:288] T 831458ca0817495dba8b124852e07d0a P bac96cb958244e309e798b2be0ef4572 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 204eff21de4d47f1aa3163d96207c97c (127.0.98.1:36429), a123c4504ef644de802def111f41624b (127.0.98.3:34921)
I20260619 16:41:04.754812 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:04.750094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36429 (local address 127.0.98.2:42379)
0619 16:41:04.750296 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:04.750315 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:04.750427 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:04.750810 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:04.750815 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:04.750840 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:04.750981 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:04.750992 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.752403 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.752407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:04.753400 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:04.753409 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.754327 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.754331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:04.754442 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:04.754468 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 16:41:04.754549 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 16:41:04.754627 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 16:41:04.754890 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:04.750358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42379 (local address 127.0.98.1:36429)
0619 16:41:04.750476 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:04.750482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:04.750496 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:41:04.750557 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:04.750560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:04.750617 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:04.750728 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:04.751136 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.752243 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.753576 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.754187 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.754273 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:04.754331 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:04.754606 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 16:41:04.754701 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:41:04.755448 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:04.750208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52369 (local address 127.0.98.3:34921)
0619 16:41:04.750396 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:04.750401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:04.750778 (+ 377us) server_negotiation.cc:407] Connection header received
0619 16:41:04.750857 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:04.750860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:04.750906 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:04.751011 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:04.751989 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.752728 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.754056 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.754685 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.754787 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:04.754852 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:04.755238 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 16:41:04.755321 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260619 16:41:04.755472 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:04.750095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34921 (local address 127.0.98.2:52369)
0619 16:41:04.750591 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:04.750605 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:04.750772 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:04.751112 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:04.751116 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:04.751136 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:04.751762 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:04.751772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.752887 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.752890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:04.753910 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:04.753920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.754866 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.754870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:04.754988 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:04.755013 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 16:41:04.755246 (+ 233us) client_negotiation.cc:240] Negotiation successful
0619 16:41:04.755321 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":55,"thread_start_us":134,"threads_started":1}
I20260619 16:41:04.755858 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831458ca0817495dba8b124852e07d0a" candidate_uuid: "bac96cb958244e309e798b2be0ef4572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "204eff21de4d47f1aa3163d96207c97c" is_pre_election: true
I20260619 16:41:04.756284 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831458ca0817495dba8b124852e07d0a" candidate_uuid: "bac96cb958244e309e798b2be0ef4572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a123c4504ef644de802def111f41624b" is_pre_election: true
I20260619 16:41:04.756443 1107 raft_consensus.cc:2454] T 831458ca0817495dba8b124852e07d0a P 204eff21de4d47f1aa3163d96207c97c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bac96cb958244e309e798b2be0ef4572 in term 0.
I20260619 16:41:04.756587 1375 raft_consensus.cc:2454] T 831458ca0817495dba8b124852e07d0a P a123c4504ef644de802def111f41624b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bac96cb958244e309e798b2be0ef4572 in term 0.
I20260619 16:41:04.757120 1176 leader_election.cc:302] T 831458ca0817495dba8b124852e07d0a P bac96cb958244e309e798b2be0ef4572 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 204eff21de4d47f1aa3163d96207c97c, a123c4504ef644de802def111f41624b, bac96cb958244e309e798b2be0ef4572; no voters:
I20260619 16:41:04.757504 1446 raft_consensus.cc:2781] T 831458ca0817495dba8b124852e07d0a P bac96cb958244e309e798b2be0ef4572 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:41:04.757699 1446 raft_consensus.cc:482] T 831458ca0817495dba8b124852e07d0a P bac96cb958244e309e798b2be0ef4572 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:41:04.757774 1446 raft_consensus.cc:3037] T 831458ca0817495dba8b124852e07d0a P bac96cb958244e309e798b2be0ef4572 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:04.759194 1446 raft_consensus.cc:504] T 831458ca0817495dba8b124852e07d0a P bac96cb958244e309e798b2be0ef4572 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac96cb958244e309e798b2be0ef4572" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38499 } } peers { permanent_uuid: "204eff21de4d47f1aa3163d96207c97c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36429 } } peers { permanent_uuid: "a123c4504ef644de802def111f41624b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34921 } }
I20260619 16:41:04.759681 1446 leader_election.cc:288] T 831458ca0817495dba8b124852e07d0a P bac96cb958244e309e798b2be0ef4572 [CANDIDATE]: Term 1 election: Requested vote from peers 204eff21de4d47f1aa3163d96207c97c (127.0.98.1:36429), a123c4504ef644de802def111f41624b (127.0.98.3:34921)
I20260619 16:41:04.760140 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831458ca0817495dba8b124852e07d0a" candidate_uuid: "bac96cb958244e309e798b2be0ef4572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "204eff21de4d47f1aa3163d96207c97c"
I20260619 16:41:04.760200 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831458ca0817495dba8b124852e07d0a" candidate_uuid: "bac96cb958244e309e798b2be0ef4572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a123c4504ef644de802def111f41624b"
I20260619 16:41:04.760267 1107 raft_consensus.cc:3037] T 831458ca0817495dba8b124852e07d0a P 204eff21de4d47f1aa3163d96207c97c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:04.760376 1375 raft_consensus.cc:3037] T 831458ca0817495dba8b124852e07d0a P a123c4504ef644de802def111f41624b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:04.761431 1107 raft_consensus.cc:2454] T 831458ca0817495dba8b124852e07d0a P 204eff21de4d47f1aa3163d96207c97c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bac96cb958244e309e798b2be0ef4572 in term 1.
I20260619 16:41:04.761528 1375 raft_consensus.cc:2454] T 831458ca0817495dba8b124852e07d0a P a123c4504ef644de802def111f41624b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bac96cb958244e309e798b2be0ef4572 in term 1.
I20260619 16:41:04.761861 1177 leader_election.cc:302] T 831458ca0817495dba8b124852e07d0a P bac96cb958244e309e798b2be0ef4572 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 204eff21de4d47f1aa3163d96207c97c, bac96cb958244e309e798b2be0ef4572; no voters:
I20260619 16:41:04.762065 1446 raft_consensus.cc:2781] T 831458ca0817495dba8b124852e07d0a P bac96cb958244e309e798b2be0ef4572 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:04.762319 1446 raft_consensus.cc:686] T 831458ca0817495dba8b124852e07d0a P bac96cb958244e309e798b2be0ef4572 [term 1 LEADER]: Becoming Leader. State: Replica: bac96cb958244e309e798b2be0ef4572, State: Running, Role: LEADER
I20260619 16:41:04.762684 1446 consensus_queue.cc:237] T 831458ca0817495dba8b124852e07d0a P bac96cb958244e309e798b2be0ef4572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac96cb958244e309e798b2be0ef4572" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38499 } } peers { permanent_uuid: "204eff21de4d47f1aa3163d96207c97c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36429 } } peers { permanent_uuid: "a123c4504ef644de802def111f41624b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34921 } }
I20260619 16:41:04.766536 962 catalog_manager.cc:5495] T 831458ca0817495dba8b124852e07d0a P bac96cb958244e309e798b2be0ef4572 reported cstate change: term changed from 0 to 1, leader changed from <none> to bac96cb958244e309e798b2be0ef4572 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bac96cb958244e309e798b2be0ef4572" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac96cb958244e309e798b2be0ef4572" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "204eff21de4d47f1aa3163d96207c97c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a123c4504ef644de802def111f41624b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34921 } health_report { overall_health: UNKNOWN } } }
W20260619 16:41:04.790686 1288 tablet.cc:2369] T 831458ca0817495dba8b124852e07d0a P bac96cb958244e309e798b2be0ef4572: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:41:04.799943 1422 tablet.cc:2369] T 831458ca0817495dba8b124852e07d0a P a123c4504ef644de802def111f41624b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:41:04.803457 1154 tablet.cc:2369] T 831458ca0817495dba8b124852e07d0a P 204eff21de4d47f1aa3163d96207c97c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 16:41:04 dist-test-slave-03cd krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:41:04.814617 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:04.805150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47990 (local address 127.0.98.1:36429)
0619 16:41:04.805313 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:04.805318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:04.805526 (+ 208us) server_negotiation.cc:407] Connection header received
0619 16:41:04.805627 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:04.805629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:04.805701 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:04.805776 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:04.806468 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.807011 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.807941 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.808156 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.808254 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:04.810944 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:04.810956 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:04.810970 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:04.810996 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:04.812737 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:04.813231 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:04.813238 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:04.813240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:04.813333 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:04.813590 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:04.813592 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:04.813593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:04.813884 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:04.814066 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:04.814260 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 16:41:04.814367 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 16:41:04 dist-test-slave-03cd krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887264, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:41:04 dist-test-slave-03cd krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887264, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:41:04.847513 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:04.835439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47992 (local address 127.0.98.1:36429)
0619 16:41:04.835602 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:04.835607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:04.835928 (+ 321us) server_negotiation.cc:407] Connection header received
0619 16:41:04.836023 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:04.836026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:04.836077 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:04.836176 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:04.837297 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.837880 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.838922 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.839159 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.839263 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:04.842680 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:04.842684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:04.842686 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:04.842724 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:04.844576 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:04.845623 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:04.845627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:04.845628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:04.845731 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:04.846188 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:04.846214 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:04.846216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:04.846505 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:04.846626 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:04.847255 (+ 629us) server_negotiation.cc:299] Negotiation successful
0619 16:41:04.847383 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 16:41:04 dist-test-slave-03cd krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:41:04 dist-test-slave-03cd krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:41:04.879530 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:04.867947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48008 (local address 127.0.98.1:36429)
0619 16:41:04.868092 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:04.868098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:04.868290 (+ 192us) server_negotiation.cc:407] Connection header received
0619 16:41:04.868519 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:04.868523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:04.868578 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:04.868681 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:04.869534 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.870083 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.871397 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:04.871644 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:04.871881 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:04.875305 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:04.875309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:04.875311 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:04.875345 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:04.877227 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:04.877800 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:04.877803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:04.877804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:04.877893 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:04.878281 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:04.878285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:04.878286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:04.878506 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:04.878627 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:04.879092 (+ 465us) server_negotiation.cc:299] Negotiation successful
0619 16:41:04.879392 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 16:41:04.881086 1067 tablet_service.cc:2908] Rejecting scan request for tablet 831458ca0817495dba8b124852e07d0a: Uninitialized: clean time has not yet been initialized
I20260619 16:41:04.883353 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 1024
I20260619 16:41:04.889762 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 1158
I20260619 16:41:04.896039 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 1292
I20260619 16:41:04.902529 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 930
2026-06-19T16:41:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2603 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:41:04 dist-test-slave-03cd krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:41:04 dist-test-slave-03cd krb5kdc[1462](info): set up 2 sockets
Jun 19 16:41:04 dist-test-slave-03cd krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260619 16:41:05.992048 392 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:41:06 dist-test-slave-03cd krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:41:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:41:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:41:06.141391 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34389
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:06.284075 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:06.284468 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:06.284518 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:06.289264 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:41:06.289372 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 16:41:06.289392 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:06.289412 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:06.289429 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:41:06.289446 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:41:06.295979 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34389
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:06.297552 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:06.298981 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:06.310518 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:06.310484 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:06.310561 1475 server_base.cc:1034] running on GCE node
W20260619 16:41:06.310451 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:06.311846 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:06.313467 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:06.314721 1475 hybrid_clock.cc:648] HybridClock initialized: now 1781887266314675 us; error 72 us; skew 500 ppm
Jun 19 16:41:06 dist-test-slave-03cd krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:06.318677 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:41:06.320159 1475 webserver.cc:460] Webserver started at http://127.0.98.62:40135/ using document root <none> and password file <none>
I20260619 16:41:06.320924 1475 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:06.321035 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:06.321324 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:06.324101 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "358abd90c863406c8e1f9e1f44c629e0"
format_stamp: "Formatted at 2026-06-19 16:41:06 on dist-test-slave-03cd"
server_key: "4927284ceb0cb4fc7f577c0fcd8e9a77"
server_key_iv: "1a5b4d3e51ff5f1fa75130818fc36971"
server_key_version: "encryptionkey@0"
I20260619 16:41:06.324813 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "358abd90c863406c8e1f9e1f44c629e0"
format_stamp: "Formatted at 2026-06-19 16:41:06 on dist-test-slave-03cd"
server_key: "4927284ceb0cb4fc7f577c0fcd8e9a77"
server_key_iv: "1a5b4d3e51ff5f1fa75130818fc36971"
server_key_version: "encryptionkey@0"
I20260619 16:41:06.330407 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 16:41:06.334143 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:06.336153 1475 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:41:06.336391 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "358abd90c863406c8e1f9e1f44c629e0"
format_stamp: "Formatted at 2026-06-19 16:41:06 on dist-test-slave-03cd"
server_key: "4927284ceb0cb4fc7f577c0fcd8e9a77"
server_key_iv: "1a5b4d3e51ff5f1fa75130818fc36971"
server_key_version: "encryptionkey@0"
I20260619 16:41:06.336557 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:06.349650 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:06.354246 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:06.354561 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:06.366029 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34389
I20260619 16:41:06.366087 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34389 every 8 connection(s)
I20260619 16:41:06.367408 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:41:06.370533 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 1475
I20260619 16:41:06.370733 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:41:06.371135 392 external_mini_cluster.cc:1428] Setting key 630d0266c1269ed6557d5625e7a4b05d
I20260619 16:41:06.371541 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:06.383359 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 358abd90c863406c8e1f9e1f44c629e0: Bootstrap starting.
Jun 19 16:41:06 dist-test-slave-03cd krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:06.386927 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 358abd90c863406c8e1f9e1f44c629e0: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:06.388603 1553 log.cc:824] T 00000000000000000000000000000000 P 358abd90c863406c8e1f9e1f44c629e0: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:06.391350 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 358abd90c863406c8e1f9e1f44c629e0: No bootstrap required, opened a new log
I20260619 16:41:06.393008 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:06.373488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56428 (local address 127.0.98.62:34389)
0619 16:41:06.374128 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:06.374155 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:06.374220 (+ 65us) server_negotiation.cc:407] Connection header received
0619 16:41:06.375087 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:06.375114 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:06.375629 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:06.376195 (+ 566us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:06.378199 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:06.379609 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:06.381148 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:06.381864 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:06.382051 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:06.385735 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:06.385744 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:06.385766 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:06.385820 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:06.388705 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:06.389592 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:06.389601 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:06.389611 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:06.389833 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:06.390207 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:06.390211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:06.390213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:06.390831 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:06.391102 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:06.391473 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 16:41:06.391936 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":221,"threads_started":1}
I20260619 16:41:06.395552 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 358abd90c863406c8e1f9e1f44c629e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "358abd90c863406c8e1f9e1f44c629e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34389 } }
I20260619 16:41:06.396005 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 358abd90c863406c8e1f9e1f44c629e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:06.396099 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 358abd90c863406c8e1f9e1f44c629e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 358abd90c863406c8e1f9e1f44c629e0, State: Initialized, Role: FOLLOWER
I20260619 16:41:06.396741 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 358abd90c863406c8e1f9e1f44c629e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "358abd90c863406c8e1f9e1f44c629e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34389 } }
I20260619 16:41:06.396953 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 358abd90c863406c8e1f9e1f44c629e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:41:06.397028 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 358abd90c863406c8e1f9e1f44c629e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:41:06.397161 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 358abd90c863406c8e1f9e1f44c629e0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:06.398257 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 358abd90c863406c8e1f9e1f44c629e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "358abd90c863406c8e1f9e1f44c629e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34389 } }
I20260619 16:41:06.398959 1553 leader_election.cc:302] T 00000000000000000000000000000000 P 358abd90c863406c8e1f9e1f44c629e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 358abd90c863406c8e1f9e1f44c629e0; no voters:
I20260619 16:41:06.399446 1553 leader_election.cc:288] T 00000000000000000000000000000000 P 358abd90c863406c8e1f9e1f44c629e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:41:06.399757 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 358abd90c863406c8e1f9e1f44c629e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:06.400177 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 358abd90c863406c8e1f9e1f44c629e0 [term 1 LEADER]: Becoming Leader. State: Replica: 358abd90c863406c8e1f9e1f44c629e0, State: Running, Role: LEADER
I20260619 16:41:06.400931 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 358abd90c863406c8e1f9e1f44c629e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "358abd90c863406c8e1f9e1f44c629e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34389 } }
I20260619 16:41:06.401496 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 358abd90c863406c8e1f9e1f44c629e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:41:06.403472 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 358abd90c863406c8e1f9e1f44c629e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "358abd90c863406c8e1f9e1f44c629e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "358abd90c863406c8e1f9e1f44c629e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34389 } } }
I20260619 16:41:06.403460 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 358abd90c863406c8e1f9e1f44c629e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 358abd90c863406c8e1f9e1f44c629e0. Latest consensus state: current_term: 1 leader_uuid: "358abd90c863406c8e1f9e1f44c629e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "358abd90c863406c8e1f9e1f44c629e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34389 } } }
I20260619 16:41:06.403769 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 358abd90c863406c8e1f9e1f44c629e0 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:06.403855 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 358abd90c863406c8e1f9e1f44c629e0 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:06.404471 1572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:41:06.408957 1572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:41:06.416868 1572 catalog_manager.cc:1347] Generated new cluster ID: 64af8bf09b054813b02b8ae969b82ae9
I20260619 16:41:06.417001 1572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:41:06.428915 1572 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:41:06.430557 1572 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:41:06.444886 1572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 358abd90c863406c8e1f9e1f44c629e0: Generated new TSK 0
I20260619 16:41:06.445963 1572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:41:06.488672 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34389
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:06.641852 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:06.642249 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:06.642390 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:06.648007 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:06.648120 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:06.648173 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:41:06.648214 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:41:06.655344 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:06.657330 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:06.658833 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:06.672906 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:06.672921 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:06.673447 1579 server_base.cc:1034] running on GCE node
W20260619 16:41:06.673806 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:06.674527 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:06.675453 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:06.676683 1579 hybrid_clock.cc:648] HybridClock initialized: now 1781887266676661 us; error 48 us; skew 500 ppm
Jun 19 16:41:06 dist-test-slave-03cd krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:06.680895 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:41:06.682155 1579 webserver.cc:460] Webserver started at http://127.0.98.1:38873/ using document root <none> and password file <none>
I20260619 16:41:06.682972 1579 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:06.683080 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:06.683380 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:06.686127 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-0/data/instance:
uuid: "1a9d14c0035140e78e58f43f8370cb6b"
format_stamp: "Formatted at 2026-06-19 16:41:06 on dist-test-slave-03cd"
server_key: "631d526c78e3f8a1c6fe4785fc3dd1c7"
server_key_iv: "f5e54b1089de603c0980f058bf6ba425"
server_key_version: "encryptionkey@0"
I20260619 16:41:06.686854 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1a9d14c0035140e78e58f43f8370cb6b"
format_stamp: "Formatted at 2026-06-19 16:41:06 on dist-test-slave-03cd"
server_key: "631d526c78e3f8a1c6fe4785fc3dd1c7"
server_key_iv: "f5e54b1089de603c0980f058bf6ba425"
server_key_version: "encryptionkey@0"
I20260619 16:41:06.691993 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 16:41:06.695910 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:06.697924 1579 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:41:06.698203 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "1a9d14c0035140e78e58f43f8370cb6b"
format_stamp: "Formatted at 2026-06-19 16:41:06 on dist-test-slave-03cd"
server_key: "631d526c78e3f8a1c6fe4785fc3dd1c7"
server_key_iv: "f5e54b1089de603c0980f058bf6ba425"
server_key_version: "encryptionkey@0"
I20260619 16:41:06.698377 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:06.726373 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:06.730331 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:06.730679 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:06.732599 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:06.732702 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:06.732777 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:06.732821 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:41:06 dist-test-slave-03cd krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:06.745323 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32805
I20260619 16:41:06.745635 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32805 every 8 connection(s)
I20260619 16:41:06.746647 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:41:06.749068 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 1579
I20260619 16:41:06.749223 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:41:06.749605 392 external_mini_cluster.cc:1428] Setting key 4937784652c9d28becd46dafd617fbed
I20260619 16:41:06.751956 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:06.733959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45063 (local address 127.0.98.62:34389)
0619 16:41:06.734161 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:06.734167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:06.735363 (+ 1196us) server_negotiation.cc:407] Connection header received
0619 16:41:06.736923 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:06.736927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:06.737009 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:06.737123 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:06.739110 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:06.739803 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:06.741762 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:06.742031 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:06.742160 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:06.745988 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:06.745993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:06.745995 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:06.746037 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:06.748250 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:06.748997 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:06.749002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:06.749004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:06.749140 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:06.749886 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:06.749891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:06.749893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:06.750256 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:06.750416 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:06.751521 (+ 1105us) server_negotiation.cc:299] Negotiation successful
0619 16:41:06.751760 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 16:41:06.753765 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:06.734533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34389 (local address 127.0.98.1:45063)
0619 16:41:06.735128 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:06.735191 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:06.736563 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:06.737361 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:06.737373 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:06.738075 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:06.738908 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:06.738928 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:06.739939 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:06.739944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:06.741520 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:06.741534 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:06.742220 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:06.742226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:06.742447 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:06.743436 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:06.743465 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:06.745710 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:06.748464 (+ 2754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:06.748470 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:06.748485 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:06.748853 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:06.749331 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:06.749337 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:06.749339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:06.749593 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:06.750481 (+ 888us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:06.750488 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:06.751109 (+ 621us) client_negotiation.cc:769] Sending connection context
0619 16:41:06.751409 (+ 300us) client_negotiation.cc:240] Negotiation successful
0619 16:41:06.751993 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":182,"threads_started":1}
I20260619 16:41:06.755913 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34389
I20260619 16:41:06.756238 1710 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:06.756930 1710 heartbeater.cc:511] Master 127.0.98.62:34389 requested a full tablet report, sending...
I20260619 16:41:06.758987 1507 ts_manager.cc:194] Registered new tserver with Master: 1a9d14c0035140e78e58f43f8370cb6b (127.0.98.1:32805)
I20260619 16:41:06.760908 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45063
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 16:41:06.769667 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:06.760264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42783 (local address 127.0.98.62:34389)
0619 16:41:06.760400 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:06.760405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:06.760541 (+ 136us) server_negotiation.cc:407] Connection header received
0619 16:41:06.760842 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:06.760846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:06.760900 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:06.761023 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:06.761882 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:06.762523 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:06.763793 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:06.764206 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:06.764309 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:06.766090 (+ 1781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:06.766096 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:06.766108 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:06.766132 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:06.767891 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:06.768360 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:06.768365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:06.768367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:06.768448 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:06.768760 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:06.768764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:06.768765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:06.768963 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:06.769075 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:06.769352 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 16:41:06.769477 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:41:06.769711 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:06.759898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34389 (local address 127.0.98.1:42783)
0619 16:41:06.760377 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:06.760404 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:06.760599 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:06.761097 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:06.761103 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:06.761438 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:06.761647 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:06.761657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:06.762698 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:06.762703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:06.763558 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:06.763577 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:06.764531 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:06.764534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:06.764671 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:06.765439 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:06.765451 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:06.765934 (+ 483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:06.768025 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:06.768029 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:06.768031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:06.768254 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:06.768553 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:06.768556 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:06.768558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:06.768653 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:06.769065 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:06.769068 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:06.769181 (+ 113us) client_negotiation.cc:769] Sending connection context
0619 16:41:06.769296 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 16:41:06.769482 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":97,"thread_start_us":146,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:41:06.788373 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34389
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:06.944726 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:06.945190 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:06.945269 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:06.950047 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:06.950182 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:06.950264 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:41:06.950341 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:41:06.957307 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:06.958967 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:06.960563 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:06.978381 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:06.978381 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:06.978658 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:06.978953 1719 server_base.cc:1034] running on GCE node
I20260619 16:41:06.980095 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:06.981727 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:06.985327 1719 hybrid_clock.cc:648] HybridClock initialized: now 1781887266985111 us; error 97 us; skew 500 ppm
Jun 19 16:41:06 dist-test-slave-03cd krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:06.990247 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:41:06.991626 1719 webserver.cc:460] Webserver started at http://127.0.98.2:34655/ using document root <none> and password file <none>
I20260619 16:41:06.992515 1719 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:06.992635 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:06.992923 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:06.995623 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-1/data/instance:
uuid: "d454b27978cb4e41abbb6e6838ddbcc8"
format_stamp: "Formatted at 2026-06-19 16:41:06 on dist-test-slave-03cd"
server_key: "3d15943342bd75fd442ffd6ae492b514"
server_key_iv: "8e411c7d1f1f3cea50777102cd496462"
server_key_version: "encryptionkey@0"
I20260619 16:41:06.996420 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d454b27978cb4e41abbb6e6838ddbcc8"
format_stamp: "Formatted at 2026-06-19 16:41:06 on dist-test-slave-03cd"
server_key: "3d15943342bd75fd442ffd6ae492b514"
server_key_iv: "8e411c7d1f1f3cea50777102cd496462"
server_key_version: "encryptionkey@0"
I20260619 16:41:07.001475 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 16:41:07.005563 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:07.007527 1719 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:41:07.007812 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "d454b27978cb4e41abbb6e6838ddbcc8"
format_stamp: "Formatted at 2026-06-19 16:41:06 on dist-test-slave-03cd"
server_key: "3d15943342bd75fd442ffd6ae492b514"
server_key_iv: "8e411c7d1f1f3cea50777102cd496462"
server_key_version: "encryptionkey@0"
I20260619 16:41:07.007979 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:07.057543 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:07.061542 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:07.061794 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:07.063915 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:07.064105 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:07.064195 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:07.064244 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:07.080072 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37281
Jun 19 16:41:07 dist-test-slave-03cd krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:07.080173 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37281 every 8 connection(s)
I20260619 16:41:07.081403 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:41:07.088068 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:07.065780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52801 (local address 127.0.98.62:34389)
0619 16:41:07.066208 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:07.066217 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:07.067820 (+ 1603us) server_negotiation.cc:407] Connection header received
0619 16:41:07.069783 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:07.069788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:07.069910 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:07.069980 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:07.072490 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.073948 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.076344 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.076730 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.076886 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:07.081262 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:07.081267 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:07.081270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:07.081311 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:07.083430 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:07.084369 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:07.084375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:07.084376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:07.084538 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:07.085939 (+ 1401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:07.085947 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:07.085952 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:07.086594 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:07.086749 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:07.087551 (+ 802us) server_negotiation.cc:299] Negotiation successful
0619 16:41:07.087872 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 16:41:07.088987 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:07.066586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34389 (local address 127.0.98.2:52801)
0619 16:41:07.067509 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:07.067580 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:07.069366 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:07.070567 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:07.070582 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:07.071273 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:07.072227 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:07.072261 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.074211 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.074215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:07.076100 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:07.076127 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.076961 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.076966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:07.077274 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:07.078278 (+ 1004us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:07.078314 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:07.080875 (+ 2561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:07.083594 (+ 2719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:07.083604 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:07.083624 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:07.083979 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:07.084698 (+ 719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:07.084701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:07.084703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:07.085626 (+ 923us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:07.086793 (+ 1167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:07.086800 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:07.087164 (+ 364us) client_negotiation.cc:769] Sending connection context
0619 16:41:07.087380 (+ 216us) client_negotiation.cc:240] Negotiation successful
0619 16:41:07.087861 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":369,"threads_started":1}
I20260619 16:41:07.091614 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 1719
I20260619 16:41:07.092070 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:41:07.092619 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34389
I20260619 16:41:07.092830 392 external_mini_cluster.cc:1428] Setting key 173fbe1968975fd76e05d740ceb89f3e
I20260619 16:41:07.092917 1850 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:07.093858 1850 heartbeater.cc:511] Master 127.0.98.62:34389 requested a full tablet report, sending...
I20260619 16:41:07.096081 1507 ts_manager.cc:194] Registered new tserver with Master: d454b27978cb4e41abbb6e6838ddbcc8 (127.0.98.2:37281)
I20260619 16:41:07.097544 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52801
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:41:07.113346 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:07.097926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34389 (local address 127.0.98.2:43739)
0619 16:41:07.099252 (+ 1326us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:07.099277 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:07.099536 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:07.100307 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:07.100312 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:07.100643 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:07.100774 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:07.100782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.101756 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.101760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:07.102999 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:07.103021 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.104441 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.104446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:07.104861 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:07.105795 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:07.105809 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:07.106238 (+ 429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:07.110873 (+ 4635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:07.110877 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:07.110880 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:07.111168 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:07.111514 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:07.111517 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:07.111520 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:07.111610 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:07.112662 (+ 1052us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:07.112665 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:07.112785 (+ 120us) client_negotiation.cc:769] Sending connection context
0619 16:41:07.112931 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 16:41:07.113131 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1120,"thread_start_us":184,"threads_started":1}
I20260619 16:41:07.113330 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:07.097987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43739 (local address 127.0.98.62:34389)
0619 16:41:07.098364 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:07.098368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:07.099421 (+ 1053us) server_negotiation.cc:407] Connection header received
0619 16:41:07.099703 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:07.099707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:07.099921 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:07.100016 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:07.100926 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.101589 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.103253 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.103854 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.104403 (+ 549us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:07.106821 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:07.106827 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:07.106830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:07.106864 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:07.110540 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:07.111303 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:07.111307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:07.111309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:07.111401 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:07.111755 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:07.111758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:07.111760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:07.112193 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:07.112666 (+ 473us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:07.112986 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 16:41:07.113126 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260619 16:41:07.395466 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.300s user 0.004s sys 0.003s
I20260619 16:41:07.396263 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34389
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:07.538358 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:07.538748 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:07.538825 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:07.543478 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:07.543591 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:07.543663 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:41:07.543787 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:41:07.549789 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:07.551086 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:07.552601 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:07.564522 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:07.564522 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:07.564522 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:07.565176 1859 server_base.cc:1034] running on GCE node
I20260619 16:41:07.565766 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:07.566716 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:07.570113 1859 hybrid_clock.cc:648] HybridClock initialized: now 1781887267570044 us; error 76 us; skew 500 ppm
Jun 19 16:41:07 dist-test-slave-03cd krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:07.574575 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:41:07.576090 1859 webserver.cc:460] Webserver started at http://127.0.98.3:43939/ using document root <none> and password file <none>
I20260619 16:41:07.576911 1859 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:07.577029 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:07.577323 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:07.580305 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-2/data/instance:
uuid: "79d241191bde448485c33b1da9698b17"
format_stamp: "Formatted at 2026-06-19 16:41:07 on dist-test-slave-03cd"
server_key: "1bb0c07b772ed4ade94ade3f393a5db9"
server_key_iv: "0a7cc33e56edfb6efa196caba572f716"
server_key_version: "encryptionkey@0"
I20260619 16:41:07.581084 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-2/wal/instance:
uuid: "79d241191bde448485c33b1da9698b17"
format_stamp: "Formatted at 2026-06-19 16:41:07 on dist-test-slave-03cd"
server_key: "1bb0c07b772ed4ade94ade3f393a5db9"
server_key_iv: "0a7cc33e56edfb6efa196caba572f716"
server_key_version: "encryptionkey@0"
I20260619 16:41:07.586361 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:41:07.590142 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:07.592777 1859 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 16:41:07.592991 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "79d241191bde448485c33b1da9698b17"
format_stamp: "Formatted at 2026-06-19 16:41:07 on dist-test-slave-03cd"
server_key: "1bb0c07b772ed4ade94ade3f393a5db9"
server_key_iv: "0a7cc33e56edfb6efa196caba572f716"
server_key_version: "encryptionkey@0"
I20260619 16:41:07.593163 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:07.607702 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:07.612196 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:07.612483 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:07.614742 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:07.614809 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:07.614861 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:07.614923 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:41:07 dist-test-slave-03cd krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:07.628679 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36617
I20260619 16:41:07.628701 1990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36617 every 8 connection(s)
I20260619 16:41:07.630133 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:41:07.633556 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:07.615948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38391 (local address 127.0.98.62:34389)
0619 16:41:07.616336 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:07.616342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:07.617267 (+ 925us) server_negotiation.cc:407] Connection header received
0619 16:41:07.618547 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:07.618553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:07.618634 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:07.618704 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:07.620617 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.621254 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.622762 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.623165 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.623294 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:07.627987 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:07.627992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:07.627995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:07.628039 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:07.630478 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:07.631393 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:07.631396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:07.631398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:07.631520 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:07.632178 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:07.632181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:07.632183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:07.632396 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:07.632490 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:07.633261 (+ 771us) server_negotiation.cc:299] Negotiation successful
0619 16:41:07.633406 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":120,"threads_started":1}
I20260619 16:41:07.634632 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:07.616526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34389 (local address 127.0.98.3:38391)
0619 16:41:07.617066 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:07.617123 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:07.618227 (+ 1104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:07.619225 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:07.619234 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:07.619833 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:07.620445 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:07.620460 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.621396 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.621400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:07.622601 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:07.622612 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.623416 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.623421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:07.623646 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:07.624540 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:07.624568 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:07.627451 (+ 2883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:07.630712 (+ 3261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:07.630722 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:07.630752 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:07.631213 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:07.631779 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:07.631783 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:07.631785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:07.631980 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:07.632525 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:07.632532 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:07.632917 (+ 385us) client_negotiation.cc:769] Sending connection context
0619 16:41:07.633114 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 16:41:07.633484 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":161,"threads_started":1}
I20260619 16:41:07.636150 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 1859
I20260619 16:41:07.636322 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:41:07.636669 392 external_mini_cluster.cc:1428] Setting key 319aea515d04fe87c360f41513107793
I20260619 16:41:07.636691 1991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34389
I20260619 16:41:07.636896 1991 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:07.637694 1991 heartbeater.cc:511] Master 127.0.98.62:34389 requested a full tablet report, sending...
I20260619 16:41:07.639880 1507 ts_manager.cc:194] Registered new tserver with Master: 79d241191bde448485c33b1da9698b17 (127.0.98.3:36617)
I20260619 16:41:07.640744 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38391
I20260619 16:41:07.650053 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:07.640145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55695 (local address 127.0.98.62:34389)
0619 16:41:07.640302 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:07.640306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:07.640322 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:41:07.640369 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:07.640371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:07.640421 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:07.640511 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:07.641157 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.641906 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.643082 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.643583 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.643763 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:07.645887 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:07.645891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:07.645895 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:07.645931 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:07.647783 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:07.648512 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:07.648515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:07.648516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:07.648613 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:07.648932 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:07.648935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:07.648936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:07.649200 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:07.649322 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:07.649718 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 16:41:07.649900 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:41:07.650216 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:07.639630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34389 (local address 127.0.98.3:55695)
0619 16:41:07.640104 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:07.640123 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:07.640232 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:07.640564 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:07.640567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:07.640831 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:07.640979 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:07.640989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.642114 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.642117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:07.642953 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:07.642963 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.644024 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.644028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:07.644135 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:07.645130 (+ 995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:07.645146 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:07.645643 (+ 497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:07.648017 (+ 2374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:07.648023 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:07.648025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:07.648359 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:07.648713 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:07.648715 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:07.648717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:07.648804 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:07.649356 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:07.649359 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:07.649487 (+ 128us) client_negotiation.cc:769] Sending connection context
0619 16:41:07.649608 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 16:41:07.649762 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":1,"thread_start_us":160,"threads_started":1}
I20260619 16:41:07.652755 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:41:07 dist-test-slave-03cd krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887267, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:41:07 dist-test-slave-03cd krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887267, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:07.686745 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:07.674635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56430 (local address 127.0.98.62:34389)
0619 16:41:07.674920 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:07.674926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:07.675126 (+ 200us) server_negotiation.cc:407] Connection header received
0619 16:41:07.675236 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:07.675240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:07.675325 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:07.675458 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:07.676762 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.677663 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.679240 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.679560 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.679744 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:07.682959 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:07.682963 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:07.682965 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:07.683008 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:07.684650 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:07.685276 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:07.685280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:07.685281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:07.685374 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:07.685680 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:07.685683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:07.685684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:07.685910 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:07.686049 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:07.686444 (+ 395us) server_negotiation.cc:299] Negotiation successful
0619 16:41:07.686594 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260619 16:41:07.691594 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:41:07.695806 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:41:07.717025 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:07.708427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36617 (local address 127.0.0.1:34476)
0619 16:41:07.709831 (+ 1404us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:07.709914 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:07.710068 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:07.710970 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:07.710979 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:07.711016 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:07.711228 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:07.711238 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.713038 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.713042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:07.715080 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:07.715087 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.716222 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.716226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:07.716391 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:07.716528 (+ 137us) client_negotiation.cc:769] Sending connection context
0619 16:41:07.716677 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 16:41:07.716893 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1156,"mutex_wait_us":540,"thread_start_us":146,"threads_started":1}
I20260619 16:41:07.717002 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:07.709993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43958 (local address 127.0.98.2:37281)
0619 16:41:07.710390 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:07.710400 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:07.710906 (+ 506us) server_negotiation.cc:407] Connection header received
0619 16:41:07.711167 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:07.711174 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:07.711458 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:07.711667 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:07.711865 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.712885 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.715212 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.716001 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.716104 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:07.716291 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:07.716721 (+ 430us) server_negotiation.cc:299] Negotiation successful
0619 16:41:07.716837 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":135,"threads_started":1}
I20260619 16:41:07.717327 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:07.708726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34476 (local address 127.0.98.3:36617)
0619 16:41:07.710276 (+ 1550us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:07.710293 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:07.710327 (+ 34us) server_negotiation.cc:407] Connection header received
0619 16:41:07.710428 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:07.710434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:07.710733 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:07.711398 (+ 665us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:07.711596 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.712861 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.715220 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.716065 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.716824 (+ 759us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:07.717002 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:07.717053 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 16:41:07.717153 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1410,"thread_start_us":196,"threads_started":1}
I20260619 16:41:07.717506 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:07.708801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32805 (local address 127.0.0.1:52440)
0619 16:41:07.710530 (+ 1729us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:07.710543 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:07.710641 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:07.711190 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:07.711194 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:07.711216 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:07.711370 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:07.711377 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.713092 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.713095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:07.714789 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:07.714800 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.716355 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.716359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:07.716559 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:07.716622 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 16:41:07.716691 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 16:41:07.717332 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1667,"mutex_wait_us":882,"thread_start_us":113,"threads_started":1}
I20260619 16:41:07.717934 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:07.709441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52440 (local address 127.0.98.1:32805)
0619 16:41:07.710395 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:07.710401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:07.710669 (+ 268us) server_negotiation.cc:407] Connection header received
0619 16:41:07.710764 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:07.710769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:07.710964 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:07.711140 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:07.711527 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.712861 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.714956 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.715940 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.716065 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:07.716295 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:07.716827 (+ 532us) server_negotiation.cc:299] Negotiation successful
0619 16:41:07.716963 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":851,"thread_start_us":129,"threads_started":1}
I20260619 16:41:07.718230 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:07.709627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37281 (local address 127.0.0.1:43958)
0619 16:41:07.710781 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:07.710792 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:07.710918 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:07.711630 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:07.711632 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:07.711649 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:07.711770 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:07.711776 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.713016 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.713020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:07.715044 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:07.715059 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.716350 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.716354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:07.716541 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:07.716600 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 16:41:07.716682 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 16:41:07.718074 (+ 1392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1112,"mutex_wait_us":351,"thread_start_us":140,"threads_started":1}
I20260619 16:41:07.721683 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 3203bfc742fd4cef80544336638053b0 (DEFAULT_TABLE table=test-table [id=8c5684a5f77d48f0b49f73979a3f59d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:07.722149 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 3203bfc742fd4cef80544336638053b0 (DEFAULT_TABLE table=test-table [id=8c5684a5f77d48f0b49f73979a3f59d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:07.722149 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 3203bfc742fd4cef80544336638053b0 (DEFAULT_TABLE table=test-table [id=8c5684a5f77d48f0b49f73979a3f59d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:07.723162 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3203bfc742fd4cef80544336638053b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:07.723605 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3203bfc742fd4cef80544336638053b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:07.723605 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3203bfc742fd4cef80544336638053b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:07.732965 2015 tablet_bootstrap.cc:492] T 3203bfc742fd4cef80544336638053b0 P 79d241191bde448485c33b1da9698b17: Bootstrap starting.
I20260619 16:41:07.733814 2017 tablet_bootstrap.cc:492] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8: Bootstrap starting.
I20260619 16:41:07.733829 2016 tablet_bootstrap.cc:492] T 3203bfc742fd4cef80544336638053b0 P 1a9d14c0035140e78e58f43f8370cb6b: Bootstrap starting.
I20260619 16:41:07.735905 2015 tablet_bootstrap.cc:654] T 3203bfc742fd4cef80544336638053b0 P 79d241191bde448485c33b1da9698b17: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:07.736099 2016 tablet_bootstrap.cc:654] T 3203bfc742fd4cef80544336638053b0 P 1a9d14c0035140e78e58f43f8370cb6b: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:07.736461 2017 tablet_bootstrap.cc:654] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:07.737071 2016 log.cc:824] T 3203bfc742fd4cef80544336638053b0 P 1a9d14c0035140e78e58f43f8370cb6b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:07.737071 2015 log.cc:824] T 3203bfc742fd4cef80544336638053b0 P 79d241191bde448485c33b1da9698b17: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:07.737648 2017 log.cc:824] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:07.739723 2015 tablet_bootstrap.cc:492] T 3203bfc742fd4cef80544336638053b0 P 79d241191bde448485c33b1da9698b17: No bootstrap required, opened a new log
I20260619 16:41:07.739859 2016 tablet_bootstrap.cc:492] T 3203bfc742fd4cef80544336638053b0 P 1a9d14c0035140e78e58f43f8370cb6b: No bootstrap required, opened a new log
I20260619 16:41:07.739969 2015 ts_tablet_manager.cc:1397] T 3203bfc742fd4cef80544336638053b0 P 79d241191bde448485c33b1da9698b17: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.005s
I20260619 16:41:07.740150 2016 ts_tablet_manager.cc:1397] T 3203bfc742fd4cef80544336638053b0 P 1a9d14c0035140e78e58f43f8370cb6b: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 16:41:07.740285 2017 tablet_bootstrap.cc:492] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8: No bootstrap required, opened a new log
I20260619 16:41:07.740492 2017 ts_tablet_manager.cc:1397] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 16:41:07.744521 2016 raft_consensus.cc:348] T 3203bfc742fd4cef80544336638053b0 P 1a9d14c0035140e78e58f43f8370cb6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } }
I20260619 16:41:07.744822 2016 raft_consensus.cc:374] T 3203bfc742fd4cef80544336638053b0 P 1a9d14c0035140e78e58f43f8370cb6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:07.744896 2016 raft_consensus.cc:729] T 3203bfc742fd4cef80544336638053b0 P 1a9d14c0035140e78e58f43f8370cb6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9d14c0035140e78e58f43f8370cb6b, State: Initialized, Role: FOLLOWER
I20260619 16:41:07.744868 2015 raft_consensus.cc:348] T 3203bfc742fd4cef80544336638053b0 P 79d241191bde448485c33b1da9698b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } }
I20260619 16:41:07.745489 2015 raft_consensus.cc:374] T 3203bfc742fd4cef80544336638053b0 P 79d241191bde448485c33b1da9698b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:07.745487 2016 consensus_queue.cc:260] T 3203bfc742fd4cef80544336638053b0 P 1a9d14c0035140e78e58f43f8370cb6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } }
I20260619 16:41:07.745509 2017 raft_consensus.cc:348] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } }
I20260619 16:41:07.745750 2015 raft_consensus.cc:729] T 3203bfc742fd4cef80544336638053b0 P 79d241191bde448485c33b1da9698b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79d241191bde448485c33b1da9698b17, State: Initialized, Role: FOLLOWER
I20260619 16:41:07.745824 2017 raft_consensus.cc:374] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:07.745926 2017 raft_consensus.cc:729] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d454b27978cb4e41abbb6e6838ddbcc8, State: Initialized, Role: FOLLOWER
I20260619 16:41:07.746435 2017 consensus_queue.cc:260] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } }
I20260619 16:41:07.746435 2015 consensus_queue.cc:260] T 3203bfc742fd4cef80544336638053b0 P 79d241191bde448485c33b1da9698b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } }
I20260619 16:41:07.746600 2016 ts_tablet_manager.cc:1428] T 3203bfc742fd4cef80544336638053b0 P 1a9d14c0035140e78e58f43f8370cb6b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 16:41:07.747116 1710 heartbeater.cc:503] Master 127.0.98.62:34389 was elected leader, sending a full tablet report...
I20260619 16:41:07.747483 1850 heartbeater.cc:503] Master 127.0.98.62:34389 was elected leader, sending a full tablet report...
I20260619 16:41:07.747680 2017 ts_tablet_manager.cc:1428] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260619 16:41:07.748644 2015 ts_tablet_manager.cc:1428] T 3203bfc742fd4cef80544336638053b0 P 79d241191bde448485c33b1da9698b17: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260619 16:41:07.749048 1991 heartbeater.cc:503] Master 127.0.98.62:34389 was elected leader, sending a full tablet report...
W20260619 16:41:07.750442 1711 tablet.cc:2369] T 3203bfc742fd4cef80544336638053b0 P 1a9d14c0035140e78e58f43f8370cb6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:41:07.833309 1851 tablet.cc:2369] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:41:07.881733 1992 tablet.cc:2369] T 3203bfc742fd4cef80544336638053b0 P 79d241191bde448485c33b1da9698b17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:41:07.897053 2022 raft_consensus.cc:482] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:41:07.897423 2022 raft_consensus.cc:504] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } }
I20260619 16:41:07.898793 2022 leader_election.cc:288] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79d241191bde448485c33b1da9698b17 (127.0.98.3:36617), 1a9d14c0035140e78e58f43f8370cb6b (127.0.98.1:32805)
I20260619 16:41:07.904067 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:07.899144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32805 (local address 127.0.98.2:39131)
0619 16:41:07.899590 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:07.899609 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:07.899772 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:07.900192 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:07.900194 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:07.900207 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:07.900358 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:07.900365 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.901144 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.901150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:07.902400 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:07.902411 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.903500 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.903503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:07.903592 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:07.903615 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:41:07.903761 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 16:41:07.903859 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":85,"thread_start_us":139,"threads_started":1}
I20260619 16:41:07.904098 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:07.899771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39131 (local address 127.0.98.1:32805)
0619 16:41:07.899917 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:07.899924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:07.899949 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:41:07.900028 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:07.900032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:07.900094 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:07.900177 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:07.900471 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.901020 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.902565 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.903169 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.903348 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:07.903441 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:07.903839 (+ 398us) server_negotiation.cc:299] Negotiation successful
0619 16:41:07.903970 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:41:07.904098 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:07.899426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39875 (local address 127.0.98.3:36617)
0619 16:41:07.899593 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:07.899599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:07.900047 (+ 448us) server_negotiation.cc:407] Connection header received
0619 16:41:07.900168 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:07.900172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:07.900239 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:07.900357 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:07.900674 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.901197 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.902723 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.903369 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.903491 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:07.903567 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:07.903861 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 16:41:07.903944 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 16:41:07.904735 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:07.899260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36617 (local address 127.0.98.2:39875)
0619 16:41:07.899885 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:07.899902 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:07.900055 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:07.900376 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:07.900380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:07.900402 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:07.900548 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:07.900554 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.901316 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.901323 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:07.902404 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:07.902415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.903509 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.903513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:07.903646 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:07.903730 (+ 84us) client_negotiation.cc:769] Sending connection context
0619 16:41:07.904508 (+ 778us) client_negotiation.cc:240] Negotiation successful
0619 16:41:07.904592 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":27,"thread_start_us":184,"threads_started":1}
I20260619 16:41:07.905190 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3203bfc742fd4cef80544336638053b0" candidate_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9d14c0035140e78e58f43f8370cb6b" is_pre_election: true
I20260619 16:41:07.905562 1664 raft_consensus.cc:2454] T 3203bfc742fd4cef80544336638053b0 P 1a9d14c0035140e78e58f43f8370cb6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d454b27978cb4e41abbb6e6838ddbcc8 in term 0.
I20260619 16:41:07.905619 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3203bfc742fd4cef80544336638053b0" candidate_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79d241191bde448485c33b1da9698b17" is_pre_election: true
I20260619 16:41:07.906107 1945 raft_consensus.cc:2454] T 3203bfc742fd4cef80544336638053b0 P 79d241191bde448485c33b1da9698b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d454b27978cb4e41abbb6e6838ddbcc8 in term 0.
I20260619 16:41:07.906137 1736 leader_election.cc:302] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9d14c0035140e78e58f43f8370cb6b, d454b27978cb4e41abbb6e6838ddbcc8; no voters:
I20260619 16:41:07.906544 2022 raft_consensus.cc:2781] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:41:07.906665 2022 raft_consensus.cc:482] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:41:07.906709 2022 raft_consensus.cc:3037] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:07.908103 2022 raft_consensus.cc:504] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } }
I20260619 16:41:07.908586 2022 leader_election.cc:288] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8 [CANDIDATE]: Term 1 election: Requested vote from peers 79d241191bde448485c33b1da9698b17 (127.0.98.3:36617), 1a9d14c0035140e78e58f43f8370cb6b (127.0.98.1:32805)
I20260619 16:41:07.909022 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3203bfc742fd4cef80544336638053b0" candidate_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79d241191bde448485c33b1da9698b17"
I20260619 16:41:07.909183 1945 raft_consensus.cc:3037] T 3203bfc742fd4cef80544336638053b0 P 79d241191bde448485c33b1da9698b17 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:07.909343 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3203bfc742fd4cef80544336638053b0" candidate_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9d14c0035140e78e58f43f8370cb6b"
I20260619 16:41:07.909699 1664 raft_consensus.cc:3037] T 3203bfc742fd4cef80544336638053b0 P 1a9d14c0035140e78e58f43f8370cb6b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:07.910750 1945 raft_consensus.cc:2454] T 3203bfc742fd4cef80544336638053b0 P 79d241191bde448485c33b1da9698b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d454b27978cb4e41abbb6e6838ddbcc8 in term 1.
I20260619 16:41:07.911013 1664 raft_consensus.cc:2454] T 3203bfc742fd4cef80544336638053b0 P 1a9d14c0035140e78e58f43f8370cb6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d454b27978cb4e41abbb6e6838ddbcc8 in term 1.
I20260619 16:41:07.911217 1737 leader_election.cc:302] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79d241191bde448485c33b1da9698b17, d454b27978cb4e41abbb6e6838ddbcc8; no voters:
I20260619 16:41:07.911489 2022 raft_consensus.cc:2781] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:07.911803 2022 raft_consensus.cc:686] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8 [term 1 LEADER]: Becoming Leader. State: Replica: d454b27978cb4e41abbb6e6838ddbcc8, State: Running, Role: LEADER
I20260619 16:41:07.912230 2022 consensus_queue.cc:237] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } }
I20260619 16:41:07.916049 1506 catalog_manager.cc:5495] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to d454b27978cb4e41abbb6e6838ddbcc8 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } health_report { overall_health: UNKNOWN } } }
I20260619 16:41:07.963759 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:07.958672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43964 (local address 127.0.98.2:37281)
0619 16:41:07.958894 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:07.958900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:07.958921 (+ 21us) server_negotiation.cc:407] Connection header received
0619 16:41:07.959185 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:07.959189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:07.959269 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:07.959402 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:41:07.959904 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.960531 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.961578 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:07.961819 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:07.961910 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:07.962319 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:41:07.962984 (+ 665us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:41:07.963091 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:07.963335 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 16:41:07.963470 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 16:41:07.973378 1664 raft_consensus.cc:1261] T 3203bfc742fd4cef80544336638053b0 P 1a9d14c0035140e78e58f43f8370cb6b [term 1 FOLLOWER]: Refusing update from remote peer d454b27978cb4e41abbb6e6838ddbcc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:07.973734 1945 raft_consensus.cc:1261] T 3203bfc742fd4cef80544336638053b0 P 79d241191bde448485c33b1da9698b17 [term 1 FOLLOWER]: Refusing update from remote peer d454b27978cb4e41abbb6e6838ddbcc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:07.974509 2026 consensus_queue.cc:1035] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:07.974771 2022 consensus_queue.cc:1035] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:08.008945 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:41:08.009272 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:41:08.014478 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 358abd90c863406c8e1f9e1f44c629e0: Sending DeleteTablet for 3 replicas of tablet 3203bfc742fd4cef80544336638053b0
I20260619 16:41:08.015941 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 3203bfc742fd4cef80544336638053b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52440
I20260619 16:41:08.015976 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 3203bfc742fd4cef80544336638053b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34476
I20260619 16:41:08.016814 2039 tablet_replica.cc:330] T 3203bfc742fd4cef80544336638053b0 P 1a9d14c0035140e78e58f43f8370cb6b: stopping tablet replica
I20260619 16:41:08.016947 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 3203bfc742fd4cef80544336638053b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43958
I20260619 16:41:08.017007 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 16:41:08.017280 2039 raft_consensus.cc:2229] T 3203bfc742fd4cef80544336638053b0 P 1a9d14c0035140e78e58f43f8370cb6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:41:08.017680 2039 raft_consensus.cc:2258] T 3203bfc742fd4cef80544336638053b0 P 1a9d14c0035140e78e58f43f8370cb6b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 16:41:08.017693 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:41:08.017943 2041 tablet_replica.cc:330] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8: stopping tablet replica
I20260619 16:41:08.018464 2041 raft_consensus.cc:2229] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:41:08.019055 2041 raft_consensus.cc:2258] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:08.019402 2039 ts_tablet_manager.cc:1905] T 3203bfc742fd4cef80544336638053b0 P 1a9d14c0035140e78e58f43f8370cb6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:08.020344 2040 tablet_replica.cc:330] T 3203bfc742fd4cef80544336638053b0 P 79d241191bde448485c33b1da9698b17: stopping tablet replica
I20260619 16:41:08.022677 2041 ts_tablet_manager.cc:1905] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:08.023754 2039 ts_tablet_manager.cc:1918] T 3203bfc742fd4cef80544336638053b0 P 1a9d14c0035140e78e58f43f8370cb6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:41:08.023934 2039 log.cc:1196] T 3203bfc742fd4cef80544336638053b0 P 1a9d14c0035140e78e58f43f8370cb6b: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-0/wal/wals/3203bfc742fd4cef80544336638053b0
I20260619 16:41:08.024291 2039 ts_tablet_manager.cc:1939] T 3203bfc742fd4cef80544336638053b0 P 1a9d14c0035140e78e58f43f8370cb6b: Deleting consensus metadata
I20260619 16:41:08.025885 1492 catalog_manager.cc:4841] TS 1a9d14c0035140e78e58f43f8370cb6b (127.0.98.1:32805): tablet 3203bfc742fd4cef80544336638053b0 (table test-table [id=8c5684a5f77d48f0b49f73979a3f59d3]) successfully deleted
I20260619 16:41:08.027788 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 7315927b389d46368ef30dbcffd827a3 (DEFAULT_TABLE table=test-table [id=69b315ed0f0c439a843d25805d2428f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:08.028167 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7315927b389d46368ef30dbcffd827a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:08.028261 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 7315927b389d46368ef30dbcffd827a3 (DEFAULT_TABLE table=test-table [id=69b315ed0f0c439a843d25805d2428f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:08.028679 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7315927b389d46368ef30dbcffd827a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:08.029587 2041 ts_tablet_manager.cc:1918] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:41:08.029722 2041 log.cc:1196] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-1/wal/wals/3203bfc742fd4cef80544336638053b0
I20260619 16:41:08.030288 2041 ts_tablet_manager.cc:1939] T 3203bfc742fd4cef80544336638053b0 P d454b27978cb4e41abbb6e6838ddbcc8: Deleting consensus metadata
I20260619 16:41:08.031363 2016 tablet_bootstrap.cc:492] T 7315927b389d46368ef30dbcffd827a3 P 1a9d14c0035140e78e58f43f8370cb6b: Bootstrap starting.
I20260619 16:41:08.032575 2016 tablet_bootstrap.cc:654] T 7315927b389d46368ef30dbcffd827a3 P 1a9d14c0035140e78e58f43f8370cb6b: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:08.033051 1493 catalog_manager.cc:4841] TS d454b27978cb4e41abbb6e6838ddbcc8 (127.0.98.2:37281): tablet 3203bfc742fd4cef80544336638053b0 (table test-table [id=8c5684a5f77d48f0b49f73979a3f59d3]) successfully deleted
I20260619 16:41:08.033334 2017 tablet_bootstrap.cc:492] T 7315927b389d46368ef30dbcffd827a3 P d454b27978cb4e41abbb6e6838ddbcc8: Bootstrap starting.
I20260619 16:41:08.028432 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 7315927b389d46368ef30dbcffd827a3 (DEFAULT_TABLE table=test-table [id=69b315ed0f0c439a843d25805d2428f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:08.034144 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7315927b389d46368ef30dbcffd827a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:08.034111 2040 raft_consensus.cc:2229] T 3203bfc742fd4cef80544336638053b0 P 79d241191bde448485c33b1da9698b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:41:08.034554 2040 raft_consensus.cc:2258] T 3203bfc742fd4cef80544336638053b0 P 79d241191bde448485c33b1da9698b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:08.034634 2017 tablet_bootstrap.cc:654] T 7315927b389d46368ef30dbcffd827a3 P d454b27978cb4e41abbb6e6838ddbcc8: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:08.036146 2016 tablet_bootstrap.cc:492] T 7315927b389d46368ef30dbcffd827a3 P 1a9d14c0035140e78e58f43f8370cb6b: No bootstrap required, opened a new log
I20260619 16:41:08.036247 2016 ts_tablet_manager.cc:1397] T 7315927b389d46368ef30dbcffd827a3 P 1a9d14c0035140e78e58f43f8370cb6b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 16:41:08.036790 2016 raft_consensus.cc:348] T 7315927b389d46368ef30dbcffd827a3 P 1a9d14c0035140e78e58f43f8370cb6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } }
I20260619 16:41:08.036944 2016 raft_consensus.cc:374] T 7315927b389d46368ef30dbcffd827a3 P 1a9d14c0035140e78e58f43f8370cb6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:08.036993 2016 raft_consensus.cc:729] T 7315927b389d46368ef30dbcffd827a3 P 1a9d14c0035140e78e58f43f8370cb6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9d14c0035140e78e58f43f8370cb6b, State: Initialized, Role: FOLLOWER
I20260619 16:41:08.037127 2016 consensus_queue.cc:260] T 7315927b389d46368ef30dbcffd827a3 P 1a9d14c0035140e78e58f43f8370cb6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } }
I20260619 16:41:08.037352 2017 tablet_bootstrap.cc:492] T 7315927b389d46368ef30dbcffd827a3 P d454b27978cb4e41abbb6e6838ddbcc8: No bootstrap required, opened a new log
I20260619 16:41:08.037388 2016 ts_tablet_manager.cc:1428] T 7315927b389d46368ef30dbcffd827a3 P 1a9d14c0035140e78e58f43f8370cb6b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 16:41:08.037460 2017 ts_tablet_manager.cc:1397] T 7315927b389d46368ef30dbcffd827a3 P d454b27978cb4e41abbb6e6838ddbcc8: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260619 16:41:08.038074 2017 raft_consensus.cc:348] T 7315927b389d46368ef30dbcffd827a3 P d454b27978cb4e41abbb6e6838ddbcc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } }
I20260619 16:41:08.038218 2017 raft_consensus.cc:374] T 7315927b389d46368ef30dbcffd827a3 P d454b27978cb4e41abbb6e6838ddbcc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:08.038250 2017 raft_consensus.cc:729] T 7315927b389d46368ef30dbcffd827a3 P d454b27978cb4e41abbb6e6838ddbcc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d454b27978cb4e41abbb6e6838ddbcc8, State: Initialized, Role: FOLLOWER
I20260619 16:41:08.038391 2017 consensus_queue.cc:260] T 7315927b389d46368ef30dbcffd827a3 P d454b27978cb4e41abbb6e6838ddbcc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } }
I20260619 16:41:08.038682 2017 ts_tablet_manager.cc:1428] T 7315927b389d46368ef30dbcffd827a3 P d454b27978cb4e41abbb6e6838ddbcc8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 16:41:08.039547 2040 ts_tablet_manager.cc:1905] T 3203bfc742fd4cef80544336638053b0 P 79d241191bde448485c33b1da9698b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:08.039968 2015 tablet_bootstrap.cc:492] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17: Bootstrap starting.
I20260619 16:41:08.041268 2015 tablet_bootstrap.cc:654] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:08.043113 2015 tablet_bootstrap.cc:492] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17: No bootstrap required, opened a new log
I20260619 16:41:08.043239 2015 ts_tablet_manager.cc:1397] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260619 16:41:08.043856 2015 raft_consensus.cc:348] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } }
I20260619 16:41:08.044085 2015 raft_consensus.cc:374] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:08.044123 2015 raft_consensus.cc:729] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79d241191bde448485c33b1da9698b17, State: Initialized, Role: FOLLOWER
I20260619 16:41:08.044267 2015 consensus_queue.cc:260] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } }
I20260619 16:41:08.044564 2015 ts_tablet_manager.cc:1428] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:41:08.044754 2040 ts_tablet_manager.cc:1918] T 3203bfc742fd4cef80544336638053b0 P 79d241191bde448485c33b1da9698b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:41:08.044888 2040 log.cc:1196] T 3203bfc742fd4cef80544336638053b0 P 79d241191bde448485c33b1da9698b17: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-2/wal/wals/3203bfc742fd4cef80544336638053b0
I20260619 16:41:08.045298 2040 ts_tablet_manager.cc:1939] T 3203bfc742fd4cef80544336638053b0 P 79d241191bde448485c33b1da9698b17: Deleting consensus metadata
I20260619 16:41:08.047227 1493 catalog_manager.cc:4841] TS 79d241191bde448485c33b1da9698b17 (127.0.98.3:36617): tablet 3203bfc742fd4cef80544336638053b0 (table test-table [id=8c5684a5f77d48f0b49f73979a3f59d3]) successfully deleted
I20260619 16:41:08.108037 2023 raft_consensus.cc:482] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:41:08.108268 2023 raft_consensus.cc:504] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } }
I20260619 16:41:08.109892 2023 leader_election.cc:288] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9d14c0035140e78e58f43f8370cb6b (127.0.98.1:32805), d454b27978cb4e41abbb6e6838ddbcc8 (127.0.98.2:37281)
I20260619 16:41:08.115456 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:08.110191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32805 (local address 127.0.98.3:49521)
0619 16:41:08.110550 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:08.110570 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:08.110704 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:08.111096 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:08.111099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:08.111119 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:08.111271 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:08.111277 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:08.112452 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:08.112456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:08.113890 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:08.113902 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:08.114925 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:08.114929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:08.115050 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:08.115078 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 16:41:08.115172 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 16:41:08.115268 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":1}
I20260619 16:41:08.115463 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:08.110376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49521 (local address 127.0.98.1:32805)
0619 16:41:08.110541 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:08.110547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:08.110813 (+ 266us) server_negotiation.cc:407] Connection header received
0619 16:41:08.110889 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:08.110892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:08.110962 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:08.111081 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:08.111390 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:08.112301 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:08.114046 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:08.114754 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:08.114882 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:08.114974 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:08.115205 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 16:41:08.115287 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:41:08.116259 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:08.110745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50699 (local address 127.0.98.2:37281)
0619 16:41:08.110898 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:08.110904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:08.110923 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:41:08.111169 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:08.111174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:08.111229 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:08.111280 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:08.111740 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:08.112516 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:08.113969 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:08.114754 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:08.114882 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:08.114976 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:08.116013 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0619 16:41:08.116129 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:41:08.116227 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7315927b389d46368ef30dbcffd827a3" candidate_uuid: "79d241191bde448485c33b1da9698b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9d14c0035140e78e58f43f8370cb6b" is_pre_election: true
I20260619 16:41:08.116432 1664 raft_consensus.cc:2454] T 7315927b389d46368ef30dbcffd827a3 P 1a9d14c0035140e78e58f43f8370cb6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79d241191bde448485c33b1da9698b17 in term 0.
I20260619 16:41:08.116703 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:08.110304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37281 (local address 127.0.98.3:50699)
0619 16:41:08.110835 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:08.110852 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:08.110999 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:08.111422 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:08.111425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:08.111442 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:08.111584 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:08.111590 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:08.112643 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:08.112650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:08.113823 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:08.113834 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:08.115714 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:08.115718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:08.115820 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:08.115844 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 16:41:08.116485 (+ 641us) client_negotiation.cc:240] Negotiation successful
0619 16:41:08.116564 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":1,"thread_start_us":98,"threads_started":1}
I20260619 16:41:08.116940 1876 leader_election.cc:302] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9d14c0035140e78e58f43f8370cb6b, 79d241191bde448485c33b1da9698b17; no voters:
I20260619 16:41:08.117398 2023 raft_consensus.cc:2781] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:41:08.117481 2023 raft_consensus.cc:482] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:41:08.117530 2023 raft_consensus.cc:3037] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:08.117513 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7315927b389d46368ef30dbcffd827a3" candidate_uuid: "79d241191bde448485c33b1da9698b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" is_pre_election: true
I20260619 16:41:08.117781 1804 raft_consensus.cc:2454] T 7315927b389d46368ef30dbcffd827a3 P d454b27978cb4e41abbb6e6838ddbcc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79d241191bde448485c33b1da9698b17 in term 0.
I20260619 16:41:08.119011 2023 raft_consensus.cc:504] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } }
I20260619 16:41:08.119510 2023 leader_election.cc:288] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17 [CANDIDATE]: Term 1 election: Requested vote from peers 1a9d14c0035140e78e58f43f8370cb6b (127.0.98.1:32805), d454b27978cb4e41abbb6e6838ddbcc8 (127.0.98.2:37281)
I20260619 16:41:08.119989 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7315927b389d46368ef30dbcffd827a3" candidate_uuid: "79d241191bde448485c33b1da9698b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9d14c0035140e78e58f43f8370cb6b"
I20260619 16:41:08.120050 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7315927b389d46368ef30dbcffd827a3" candidate_uuid: "79d241191bde448485c33b1da9698b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d454b27978cb4e41abbb6e6838ddbcc8"
I20260619 16:41:08.120188 1664 raft_consensus.cc:3037] T 7315927b389d46368ef30dbcffd827a3 P 1a9d14c0035140e78e58f43f8370cb6b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:08.120216 1804 raft_consensus.cc:3037] T 7315927b389d46368ef30dbcffd827a3 P d454b27978cb4e41abbb6e6838ddbcc8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:08.121388 1804 raft_consensus.cc:2454] T 7315927b389d46368ef30dbcffd827a3 P d454b27978cb4e41abbb6e6838ddbcc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79d241191bde448485c33b1da9698b17 in term 1.
I20260619 16:41:08.121646 1664 raft_consensus.cc:2454] T 7315927b389d46368ef30dbcffd827a3 P 1a9d14c0035140e78e58f43f8370cb6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79d241191bde448485c33b1da9698b17 in term 1.
I20260619 16:41:08.121800 1877 leader_election.cc:302] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79d241191bde448485c33b1da9698b17, d454b27978cb4e41abbb6e6838ddbcc8; no voters:
I20260619 16:41:08.122212 2023 raft_consensus.cc:2781] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:08.122458 2023 raft_consensus.cc:686] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17 [term 1 LEADER]: Becoming Leader. State: Replica: 79d241191bde448485c33b1da9698b17, State: Running, Role: LEADER
I20260619 16:41:08.122776 2023 consensus_queue.cc:237] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } }
I20260619 16:41:08.126091 1507 catalog_manager.cc:5495] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79d241191bde448485c33b1da9698b17 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "79d241191bde448485c33b1da9698b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } health_report { overall_health: HEALTHY } } }
Jun 19 16:41:08 dist-test-slave-03cd krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:08.174374 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:08.162199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34389 (local address 127.0.0.1:56432)
0619 16:41:08.162699 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:08.162721 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:08.162904 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:08.163315 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:08.163319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:08.164002 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:08.164160 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:08.164169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:08.165265 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:08.165269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:08.166212 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:08.166223 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:08.166881 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:08.166886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:08.166986 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:08.167958 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:08.167975 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:08.169984 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:08.172386 (+ 2402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:08.172406 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:08.172420 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:08.172722 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:08.173102 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:08.173105 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:08.173107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:08.173192 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:08.173657 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:08.173670 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:08.173807 (+ 137us) client_negotiation.cc:769] Sending connection context
0619 16:41:08.173941 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 16:41:08.174231 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":105,"thread_start_us":171,"threads_started":1}
I20260619 16:41:08.174372 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:08.162382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56432 (local address 127.0.98.62:34389)
0619 16:41:08.162530 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:08.162535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:08.162888 (+ 353us) server_negotiation.cc:407] Connection header received
0619 16:41:08.163067 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:08.163070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:08.163132 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:08.163234 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:08.164371 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:08.165090 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:08.166397 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:08.166705 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:08.166829 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:08.170190 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:08.170194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:08.170196 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:08.170220 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:08.172218 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:08.172855 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:08.172862 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:08.172864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:08.172951 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:08.173299 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:08.173305 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:08.173307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:08.173535 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:08.173656 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:08.173985 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 16:41:08.174155 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:41:08.180888 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56432:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 16:41:08.182003 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:41:08.189496 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 58c2061f8dd54e6d957807664187e735 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b0f1d02402874e498b065378adef4311]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:41:08.189630 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 58c2061f8dd54e6d957807664187e735 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b0f1d02402874e498b065378adef4311]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:41:08.189594 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 58c2061f8dd54e6d957807664187e735 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b0f1d02402874e498b065378adef4311]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:41:08.190060 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58c2061f8dd54e6d957807664187e735. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:08.190062 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58c2061f8dd54e6d957807664187e735. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:08.190065 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58c2061f8dd54e6d957807664187e735. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:08.193971 2017 tablet_bootstrap.cc:492] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8: Bootstrap starting.
I20260619 16:41:08.194432 2015 tablet_bootstrap.cc:492] T 58c2061f8dd54e6d957807664187e735 P 79d241191bde448485c33b1da9698b17: Bootstrap starting.
I20260619 16:41:08.195156 2017 tablet_bootstrap.cc:654] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:08.195622 2015 tablet_bootstrap.cc:654] T 58c2061f8dd54e6d957807664187e735 P 79d241191bde448485c33b1da9698b17: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:08.196794 2016 tablet_bootstrap.cc:492] T 58c2061f8dd54e6d957807664187e735 P 1a9d14c0035140e78e58f43f8370cb6b: Bootstrap starting.
I20260619 16:41:08.197387 2017 tablet_bootstrap.cc:492] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8: No bootstrap required, opened a new log
I20260619 16:41:08.197472 2017 ts_tablet_manager.cc:1397] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260619 16:41:08.197880 2016 tablet_bootstrap.cc:654] T 58c2061f8dd54e6d957807664187e735 P 1a9d14c0035140e78e58f43f8370cb6b: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:08.198041 2017 raft_consensus.cc:348] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } }
I20260619 16:41:08.198202 2017 raft_consensus.cc:374] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:08.198230 2017 raft_consensus.cc:729] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d454b27978cb4e41abbb6e6838ddbcc8, State: Initialized, Role: FOLLOWER
I20260619 16:41:08.198380 2017 consensus_queue.cc:260] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } }
I20260619 16:41:08.198702 2017 ts_tablet_manager.cc:1428] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 16:41:08.198652 2026 tablet_replica.cc:439] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } } }
I20260619 16:41:08.198798 2026 tablet_replica.cc:442] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:41:08.200029 2015 tablet_bootstrap.cc:492] T 58c2061f8dd54e6d957807664187e735 P 79d241191bde448485c33b1da9698b17: No bootstrap required, opened a new log
I20260619 16:41:08.200145 2015 ts_tablet_manager.cc:1397] T 58c2061f8dd54e6d957807664187e735 P 79d241191bde448485c33b1da9698b17: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260619 16:41:08.200623 2015 raft_consensus.cc:348] T 58c2061f8dd54e6d957807664187e735 P 79d241191bde448485c33b1da9698b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } }
I20260619 16:41:08.200817 2015 raft_consensus.cc:374] T 58c2061f8dd54e6d957807664187e735 P 79d241191bde448485c33b1da9698b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:08.200855 2015 raft_consensus.cc:729] T 58c2061f8dd54e6d957807664187e735 P 79d241191bde448485c33b1da9698b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79d241191bde448485c33b1da9698b17, State: Initialized, Role: FOLLOWER
I20260619 16:41:08.201040 2015 consensus_queue.cc:260] T 58c2061f8dd54e6d957807664187e735 P 79d241191bde448485c33b1da9698b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } }
I20260619 16:41:08.201304 2023 tablet_replica.cc:439] T 58c2061f8dd54e6d957807664187e735 P 79d241191bde448485c33b1da9698b17: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } } }
I20260619 16:41:08.201455 2023 tablet_replica.cc:442] T 58c2061f8dd54e6d957807664187e735 P 79d241191bde448485c33b1da9698b17: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:41:08.202301 2016 tablet_bootstrap.cc:492] T 58c2061f8dd54e6d957807664187e735 P 1a9d14c0035140e78e58f43f8370cb6b: No bootstrap required, opened a new log
I20260619 16:41:08.202399 2016 ts_tablet_manager.cc:1397] T 58c2061f8dd54e6d957807664187e735 P 1a9d14c0035140e78e58f43f8370cb6b: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260619 16:41:08.202539 2015 ts_tablet_manager.cc:1428] T 58c2061f8dd54e6d957807664187e735 P 79d241191bde448485c33b1da9698b17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:41:08.203047 2016 raft_consensus.cc:348] T 58c2061f8dd54e6d957807664187e735 P 1a9d14c0035140e78e58f43f8370cb6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } }
I20260619 16:41:08.203173 2016 raft_consensus.cc:374] T 58c2061f8dd54e6d957807664187e735 P 1a9d14c0035140e78e58f43f8370cb6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:08.203217 2016 raft_consensus.cc:729] T 58c2061f8dd54e6d957807664187e735 P 1a9d14c0035140e78e58f43f8370cb6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9d14c0035140e78e58f43f8370cb6b, State: Initialized, Role: FOLLOWER
I20260619 16:41:08.203363 2016 consensus_queue.cc:260] T 58c2061f8dd54e6d957807664187e735 P 1a9d14c0035140e78e58f43f8370cb6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } }
I20260619 16:41:08.203650 2021 tablet_replica.cc:439] T 58c2061f8dd54e6d957807664187e735 P 1a9d14c0035140e78e58f43f8370cb6b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } } }
I20260619 16:41:08.203811 2021 tablet_replica.cc:442] T 58c2061f8dd54e6d957807664187e735 P 1a9d14c0035140e78e58f43f8370cb6b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:41:08.204263 2016 ts_tablet_manager.cc:1428] T 58c2061f8dd54e6d957807664187e735 P 1a9d14c0035140e78e58f43f8370cb6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:41:08.321633 2026 raft_consensus.cc:482] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:41:08.321894 2026 raft_consensus.cc:504] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } }
I20260619 16:41:08.322490 2026 leader_election.cc:288] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9d14c0035140e78e58f43f8370cb6b (127.0.98.1:32805), 79d241191bde448485c33b1da9698b17 (127.0.98.3:36617)
I20260619 16:41:08.323268 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c2061f8dd54e6d957807664187e735" candidate_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9d14c0035140e78e58f43f8370cb6b" is_pre_election: true
I20260619 16:41:08.323268 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c2061f8dd54e6d957807664187e735" candidate_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79d241191bde448485c33b1da9698b17" is_pre_election: true
I20260619 16:41:08.323484 1945 raft_consensus.cc:2454] T 58c2061f8dd54e6d957807664187e735 P 79d241191bde448485c33b1da9698b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d454b27978cb4e41abbb6e6838ddbcc8 in term 0.
I20260619 16:41:08.323484 1664 raft_consensus.cc:2454] T 58c2061f8dd54e6d957807664187e735 P 1a9d14c0035140e78e58f43f8370cb6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d454b27978cb4e41abbb6e6838ddbcc8 in term 0.
I20260619 16:41:08.323951 1737 leader_election.cc:302] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79d241191bde448485c33b1da9698b17, d454b27978cb4e41abbb6e6838ddbcc8; no voters:
I20260619 16:41:08.324193 2026 raft_consensus.cc:2781] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:41:08.324260 2026 raft_consensus.cc:482] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:41:08.324292 2026 raft_consensus.cc:3037] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:08.325623 2026 raft_consensus.cc:504] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } }
I20260619 16:41:08.326087 2026 leader_election.cc:288] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8 [CANDIDATE]: Term 1 election: Requested vote from peers 1a9d14c0035140e78e58f43f8370cb6b (127.0.98.1:32805), 79d241191bde448485c33b1da9698b17 (127.0.98.3:36617)
I20260619 16:41:08.326579 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c2061f8dd54e6d957807664187e735" candidate_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79d241191bde448485c33b1da9698b17"
I20260619 16:41:08.326769 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c2061f8dd54e6d957807664187e735" candidate_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9d14c0035140e78e58f43f8370cb6b"
I20260619 16:41:08.326838 1945 raft_consensus.cc:3037] T 58c2061f8dd54e6d957807664187e735 P 79d241191bde448485c33b1da9698b17 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:08.326943 1664 raft_consensus.cc:3037] T 58c2061f8dd54e6d957807664187e735 P 1a9d14c0035140e78e58f43f8370cb6b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:08.328150 1945 raft_consensus.cc:2454] T 58c2061f8dd54e6d957807664187e735 P 79d241191bde448485c33b1da9698b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d454b27978cb4e41abbb6e6838ddbcc8 in term 1.
I20260619 16:41:08.328367 1664 raft_consensus.cc:2454] T 58c2061f8dd54e6d957807664187e735 P 1a9d14c0035140e78e58f43f8370cb6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d454b27978cb4e41abbb6e6838ddbcc8 in term 1.
I20260619 16:41:08.328547 1737 leader_election.cc:302] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79d241191bde448485c33b1da9698b17, d454b27978cb4e41abbb6e6838ddbcc8; no voters:
I20260619 16:41:08.328842 2026 raft_consensus.cc:2781] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:08.328971 2026 raft_consensus.cc:686] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8 [term 1 LEADER]: Becoming Leader. State: Replica: d454b27978cb4e41abbb6e6838ddbcc8, State: Running, Role: LEADER
I20260619 16:41:08.329175 2026 consensus_queue.cc:237] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } }
I20260619 16:41:08.330466 2035 tablet_replica.cc:439] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8: TxnStatusTablet state changed. Reason: New leader d454b27978cb4e41abbb6e6838ddbcc8. Latest consensus state: current_term: 1 leader_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } } }
I20260619 16:41:08.330616 2035 tablet_replica.cc:442] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:41:08.330963 2058 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 16:41:08.331168 2058 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 16:41:08.331362 1506 catalog_manager.cc:5495] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to d454b27978cb4e41abbb6e6838ddbcc8 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } health_report { overall_health: UNKNOWN } } }
I20260619 16:41:08.373095 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:08.367990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37281 (local address 127.0.0.1:43970)
0619 16:41:08.368308 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:08.368325 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:08.368481 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:08.368894 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:08.368897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:08.368909 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:41:08.369039 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:08.369045 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:08.370207 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:08.370211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:08.371251 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:08.371267 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:08.372043 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:08.372048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:08.372156 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:08.372233 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:41:08.372666 (+ 433us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:41:08.372684 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 16:41:08.372764 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 16:41:08.372891 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198}
I20260619 16:41:08.373222 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:08.368134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43970 (local address 127.0.98.2:37281)
0619 16:41:08.368289 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:08.368295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:08.368415 (+ 120us) server_negotiation.cc:407] Connection header received
0619 16:41:08.368641 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:08.368644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:08.368716 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:08.368848 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:41:08.369172 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:08.370033 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:08.371439 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:08.371902 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:08.372019 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:08.372446 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:41:08.372565 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:41:08.372643 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:08.372967 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 16:41:08.373045 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:41:08.377256 1664 raft_consensus.cc:1261] T 58c2061f8dd54e6d957807664187e735 P 1a9d14c0035140e78e58f43f8370cb6b [term 1 FOLLOWER]: Refusing update from remote peer d454b27978cb4e41abbb6e6838ddbcc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:08.377332 1945 raft_consensus.cc:1261] T 58c2061f8dd54e6d957807664187e735 P 79d241191bde448485c33b1da9698b17 [term 1 FOLLOWER]: Refusing update from remote peer d454b27978cb4e41abbb6e6838ddbcc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:08.378147 2026 consensus_queue.cc:1035] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:08.378347 2035 consensus_queue.cc:1035] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:08.381069 2021 tablet_replica.cc:439] T 58c2061f8dd54e6d957807664187e735 P 1a9d14c0035140e78e58f43f8370cb6b: TxnStatusTablet state changed. Reason: New leader d454b27978cb4e41abbb6e6838ddbcc8. Latest consensus state: current_term: 1 leader_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } } }
I20260619 16:41:08.381203 2023 tablet_replica.cc:439] T 58c2061f8dd54e6d957807664187e735 P 79d241191bde448485c33b1da9698b17: TxnStatusTablet state changed. Reason: New leader d454b27978cb4e41abbb6e6838ddbcc8. Latest consensus state: current_term: 1 leader_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } } }
I20260619 16:41:08.381301 2021 tablet_replica.cc:442] T 58c2061f8dd54e6d957807664187e735 P 1a9d14c0035140e78e58f43f8370cb6b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:41:08.381331 2023 tablet_replica.cc:442] T 58c2061f8dd54e6d957807664187e735 P 79d241191bde448485c33b1da9698b17: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:41:08.384613 2026 tablet_replica.cc:439] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } } }
I20260619 16:41:08.384819 2026 tablet_replica.cc:442] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:41:08.385298 2029 mvcc.cc:205] Tried to move back new op lower bound from 7298610251267493888 to 7298610251077992448. Current Snapshot: MvccSnapshot[applied={T|T < 7298610251267493888 or (T in {7298610251267493888})}]
I20260619 16:41:08.386914 2031 mvcc.cc:205] Tried to move back new op lower bound from 7298610251267493888 to 7298610251077992448. Current Snapshot: MvccSnapshot[applied={T|T < 7298610251267493888}]
I20260619 16:41:08.387173 2021 tablet_replica.cc:439] T 58c2061f8dd54e6d957807664187e735 P 1a9d14c0035140e78e58f43f8370cb6b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } } }
I20260619 16:41:08.387321 2021 tablet_replica.cc:442] T 58c2061f8dd54e6d957807664187e735 P 1a9d14c0035140e78e58f43f8370cb6b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:41:08.389698 2022 tablet_replica.cc:439] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } } }
I20260619 16:41:08.389873 2022 tablet_replica.cc:442] T 58c2061f8dd54e6d957807664187e735 P d454b27978cb4e41abbb6e6838ddbcc8: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:41:08.391016 2030 mvcc.cc:205] Tried to move back new op lower bound from 7298610251267493888 to 7298610251077992448. Current Snapshot: MvccSnapshot[applied={T|T < 7298610251267493888 or (T in {7298610251267493888})}]
I20260619 16:41:08.391389 2023 tablet_replica.cc:439] T 58c2061f8dd54e6d957807664187e735 P 79d241191bde448485c33b1da9698b17: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 } } peers { permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 } } peers { permanent_uuid: "79d241191bde448485c33b1da9698b17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36617 } } }
I20260619 16:41:08.391539 2023 tablet_replica.cc:442] T 58c2061f8dd54e6d957807664187e735 P 79d241191bde448485c33b1da9698b17: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:41:08.397166 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:08.391869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34492 (local address 127.0.98.3:36617)
0619 16:41:08.392386 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:08.392390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:08.392403 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:41:08.392482 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:08.392484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:08.392548 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:08.392656 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:41:08.393246 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:08.393993 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:08.395228 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:08.395499 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:08.395644 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:08.396016 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:41:08.396667 (+ 651us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:41:08.396823 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:08.396896 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 16:41:08.396974 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419}
I20260619 16:41:08.413405 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:08.408149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53105 (local address 127.0.98.2:37281)
0619 16:41:08.408350 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:08.408356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:08.408635 (+ 279us) server_negotiation.cc:407] Connection header received
0619 16:41:08.408792 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:08.408795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:08.408857 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:08.408956 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:41:08.409277 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:08.409871 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:08.412035 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:08.412314 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:08.412425 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:08.412800 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:41:08.412907 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:41:08.412969 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:08.413134 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 16:41:08.413241 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 16:41:08.413483 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:08.408027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37281 (local address 127.0.98.3:53105)
0619 16:41:08.408464 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:08.408485 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:08.408657 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:08.408995 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:08.408997 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:08.409006 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:41:08.409148 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:08.409154 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:08.410021 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:08.410025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:08.411842 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:08.411857 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:08.412447 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:08.412451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:08.412558 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:08.412630 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:41:08.413025 (+ 395us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:41:08.413032 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 16:41:08.413111 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 16:41:08.413231 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":172,"threads_started":1}
I20260619 16:41:08.423293 1804 raft_consensus.cc:1261] T 7315927b389d46368ef30dbcffd827a3 P d454b27978cb4e41abbb6e6838ddbcc8 [term 1 FOLLOWER]: Refusing update from remote peer 79d241191bde448485c33b1da9698b17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:08.423915 1664 raft_consensus.cc:1261] T 7315927b389d46368ef30dbcffd827a3 P 1a9d14c0035140e78e58f43f8370cb6b [term 1 FOLLOWER]: Refusing update from remote peer 79d241191bde448485c33b1da9698b17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:08.424186 2046 consensus_queue.cc:1035] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d454b27978cb4e41abbb6e6838ddbcc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:08.424445 2023 consensus_queue.cc:1035] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9d14c0035140e78e58f43f8370cb6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:08.452827 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:08.447973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41717 (local address 127.0.98.3:36617)
0619 16:41:08.448144 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:08.448149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:08.448497 (+ 348us) server_negotiation.cc:407] Connection header received
0619 16:41:08.448638 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:08.448645 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:08.448718 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:08.448861 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:41:08.449206 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:08.450055 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:08.451284 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:08.451719 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:08.451858 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:08.452196 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:41:08.452305 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:41:08.452402 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:08.452590 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 16:41:08.452662 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:41:08.452910 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:08.447864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36617 (local address 127.0.98.2:41717)
0619 16:41:08.448373 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:08.448390 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:08.448527 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:08.448877 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:08.448881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:08.448892 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:41:08.449036 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:08.449043 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:08.450203 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:08.450207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:08.451115 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:08.451126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:08.451858 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:08.451861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:08.451936 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:08.452003 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:41:08.452454 (+ 451us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:41:08.452463 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 16:41:08.452557 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 16:41:08.452695 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":138,"threads_started":1}
I20260619 16:41:08.488797 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:41:08.488996 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:41:08.491421 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 358abd90c863406c8e1f9e1f44c629e0: Sending DeleteTablet for 3 replicas of tablet 7315927b389d46368ef30dbcffd827a3
I20260619 16:41:08.492412 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 7315927b389d46368ef30dbcffd827a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43958
I20260619 16:41:08.492609 2041 tablet_replica.cc:330] T 7315927b389d46368ef30dbcffd827a3 P d454b27978cb4e41abbb6e6838ddbcc8: stopping tablet replica
I20260619 16:41:08.492646 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 7315927b389d46368ef30dbcffd827a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52440
I20260619 16:41:08.492725 2041 raft_consensus.cc:2229] T 7315927b389d46368ef30dbcffd827a3 P d454b27978cb4e41abbb6e6838ddbcc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:41:08.492861 2041 raft_consensus.cc:2258] T 7315927b389d46368ef30dbcffd827a3 P d454b27978cb4e41abbb6e6838ddbcc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:08.492967 2039 tablet_replica.cc:330] T 7315927b389d46368ef30dbcffd827a3 P 1a9d14c0035140e78e58f43f8370cb6b: stopping tablet replica
I20260619 16:41:08.493117 2039 raft_consensus.cc:2229] T 7315927b389d46368ef30dbcffd827a3 P 1a9d14c0035140e78e58f43f8370cb6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:41:08.493278 2039 raft_consensus.cc:2258] T 7315927b389d46368ef30dbcffd827a3 P 1a9d14c0035140e78e58f43f8370cb6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:08.493708 2041 ts_tablet_manager.cc:1905] T 7315927b389d46368ef30dbcffd827a3 P d454b27978cb4e41abbb6e6838ddbcc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:08.494632 2039 ts_tablet_manager.cc:1905] T 7315927b389d46368ef30dbcffd827a3 P 1a9d14c0035140e78e58f43f8370cb6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:08.496999 2041 ts_tablet_manager.cc:1918] T 7315927b389d46368ef30dbcffd827a3 P d454b27978cb4e41abbb6e6838ddbcc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:41:08.497120 2041 log.cc:1196] T 7315927b389d46368ef30dbcffd827a3 P d454b27978cb4e41abbb6e6838ddbcc8: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-1/wal/wals/7315927b389d46368ef30dbcffd827a3
I20260619 16:41:08.497437 2041 ts_tablet_manager.cc:1939] T 7315927b389d46368ef30dbcffd827a3 P d454b27978cb4e41abbb6e6838ddbcc8: Deleting consensus metadata
I20260619 16:41:08.498093 2039 ts_tablet_manager.cc:1918] T 7315927b389d46368ef30dbcffd827a3 P 1a9d14c0035140e78e58f43f8370cb6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:41:08.498284 1493 catalog_manager.cc:4841] TS d454b27978cb4e41abbb6e6838ddbcc8 (127.0.98.2:37281): tablet 7315927b389d46368ef30dbcffd827a3 (table test-table [id=69b315ed0f0c439a843d25805d2428f3]) successfully deleted
I20260619 16:41:08.499130 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 7315927b389d46368ef30dbcffd827a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34476
I20260619 16:41:08.499366 2040 tablet_replica.cc:330] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17: stopping tablet replica
I20260619 16:41:08.499486 2040 raft_consensus.cc:2229] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:41:08.499894 2040 raft_consensus.cc:2258] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:08.500731 2039 log.cc:1196] T 7315927b389d46368ef30dbcffd827a3 P 1a9d14c0035140e78e58f43f8370cb6b: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-0/wal/wals/7315927b389d46368ef30dbcffd827a3
I20260619 16:41:08.500787 2040 ts_tablet_manager.cc:1905] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:08.501147 2039 ts_tablet_manager.cc:1939] T 7315927b389d46368ef30dbcffd827a3 P 1a9d14c0035140e78e58f43f8370cb6b: Deleting consensus metadata
I20260619 16:41:08.502038 1492 catalog_manager.cc:4841] TS 1a9d14c0035140e78e58f43f8370cb6b (127.0.98.1:32805): tablet 7315927b389d46368ef30dbcffd827a3 (table test-table [id=69b315ed0f0c439a843d25805d2428f3]) successfully deleted
I20260619 16:41:08.505079 2040 ts_tablet_manager.cc:1918] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:41:08.505265 2040 log.cc:1196] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887259574980-392-0/minicluster-data/ts-2/wal/wals/7315927b389d46368ef30dbcffd827a3
I20260619 16:41:08.505612 2040 ts_tablet_manager.cc:1939] T 7315927b389d46368ef30dbcffd827a3 P 79d241191bde448485c33b1da9698b17: Deleting consensus metadata
I20260619 16:41:08.506639 1493 catalog_manager.cc:4841] TS 79d241191bde448485c33b1da9698b17 (127.0.98.3:36617): tablet 7315927b389d46368ef30dbcffd827a3 (table test-table [id=69b315ed0f0c439a843d25805d2428f3]) successfully deleted
Jun 19 16:41:08 dist-test-slave-03cd krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887267, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:41:08.517760 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:08.502749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52446 (local address 127.0.98.1:32805)
0619 16:41:08.502882 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:08.502888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:08.503705 (+ 817us) server_negotiation.cc:407] Connection header received
0619 16:41:08.503976 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:08.503980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:08.504044 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:08.504131 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:08.505265 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:08.506122 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:08.507455 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:08.507929 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:08.508095 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:08.512774 (+ 4679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:08.512815 (+ 41us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:08.512838 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:08.512888 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:08.515288 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:08.516021 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:08.516028 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:08.516030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:08.516147 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:08.516515 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:08.516518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:08.516520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:08.516843 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:08.517087 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:08.517399 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 16:41:08.517557 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 16:41:08.519271 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52446
I20260619 16:41:08.533248 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:08.521880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56436 (local address 127.0.98.62:34389)
0619 16:41:08.522058 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:08.522063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:08.522092 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:41:08.522230 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:08.522234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:08.522300 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:08.522416 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:08.523256 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:08.524297 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:08.525445 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:08.525909 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:08.526050 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:08.527828 (+ 1778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:08.527834 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:08.527837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:08.527876 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:08.530827 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:08.531525 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:08.531529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:08.531532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:08.531717 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:08.532090 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:08.532093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:08.532096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:08.532403 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:08.532552 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:08.532855 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 16:41:08.533028 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260619 16:41:08.533854 1507 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56436
I20260619 16:41:08.536181 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 1579
I20260619 16:41:08.545521 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 1719
I20260619 16:41:08.555704 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 1859
I20260619 16:41:08.563975 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 1475
2026-06-19T16:41:08Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3663 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:41:08 dist-test-slave-03cd krb5kdc[2082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:41:08 dist-test-slave-03cd krb5kdc[2082](info): set up 2 sockets
Jun 19 16:41:08 dist-test-slave-03cd krb5kdc[2082](info): commencing operation
krb5kdc: starting...
W20260619 16:41:09.688452 392 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:41:09 dist-test-slave-03cd krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:41:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:41:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:41:09.828039 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38541
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:09.955478 2095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:09.955852 2095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:09.955906 2095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:09.960427 2095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:41:09.960511 2095 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 16:41:09.960529 2095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:09.960548 2095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:09.960565 2095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:41:09.960615 2095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:41:09.967417 2095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38541
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:09.968690 2095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:09.969835 2095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:09.983819 2103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:09.983819 2101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:09.984306 2100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:09.984850 2095 server_base.cc:1034] running on GCE node
I20260619 16:41:09.985914 2095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:09.987637 2095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:09.989020 2095 hybrid_clock.cc:648] HybridClock initialized: now 1781887269988956 us; error 70 us; skew 500 ppm
Jun 19 16:41:09 dist-test-slave-03cd krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:09.998660 2095 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:41:10.000010 2095 webserver.cc:460] Webserver started at http://127.0.98.62:41571/ using document root <none> and password file <none>
I20260619 16:41:10.000698 2095 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:10.000777 2095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:10.001039 2095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:10.003473 2095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "2e74dc107d0148a4a928ae1bce521b71"
format_stamp: "Formatted at 2026-06-19 16:41:09 on dist-test-slave-03cd"
server_key: "f107a31dad0192b714a05b01bf3b6f46"
server_key_iv: "4d1e64116e3039a967ffbbb43169837e"
server_key_version: "encryptionkey@0"
I20260619 16:41:10.004236 2095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "2e74dc107d0148a4a928ae1bce521b71"
format_stamp: "Formatted at 2026-06-19 16:41:09 on dist-test-slave-03cd"
server_key: "f107a31dad0192b714a05b01bf3b6f46"
server_key_iv: "4d1e64116e3039a967ffbbb43169837e"
server_key_version: "encryptionkey@0"
I20260619 16:41:10.009156 2095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:41:10.012346 2110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:10.014525 2095 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260619 16:41:10.014745 2095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "2e74dc107d0148a4a928ae1bce521b71"
format_stamp: "Formatted at 2026-06-19 16:41:09 on dist-test-slave-03cd"
server_key: "f107a31dad0192b714a05b01bf3b6f46"
server_key_iv: "4d1e64116e3039a967ffbbb43169837e"
server_key_version: "encryptionkey@0"
I20260619 16:41:10.014910 2095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:10.037390 2095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:10.041257 2095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:10.041513 2095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:10.051103 2095 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38541
I20260619 16:41:10.051154 2172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38541 every 8 connection(s)
I20260619 16:41:10.052306 2095 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:41:10.055447 2173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:10.056433 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 2095
I20260619 16:41:10.056565 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:41:10.056860 392 external_mini_cluster.cc:1428] Setting key db2d8937872bb89d3e8a712b9511456c
I20260619 16:41:10.064452 2173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e74dc107d0148a4a928ae1bce521b71: Bootstrap starting.
I20260619 16:41:10.067134 2173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e74dc107d0148a4a928ae1bce521b71: Neither blocks nor log segments found. Creating new log.
Jun 19 16:41:10 dist-test-slave-03cd krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:10.068229 2173 log.cc:824] T 00000000000000000000000000000000 P 2e74dc107d0148a4a928ae1bce521b71: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:10.070840 2173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e74dc107d0148a4a928ae1bce521b71: No bootstrap required, opened a new log
I20260619 16:41:10.074163 2173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e74dc107d0148a4a928ae1bce521b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e74dc107d0148a4a928ae1bce521b71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } }
I20260619 16:41:10.074515 2173 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e74dc107d0148a4a928ae1bce521b71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:10.074601 2173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e74dc107d0148a4a928ae1bce521b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e74dc107d0148a4a928ae1bce521b71, State: Initialized, Role: FOLLOWER
I20260619 16:41:10.075140 2173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e74dc107d0148a4a928ae1bce521b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e74dc107d0148a4a928ae1bce521b71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } }
I20260619 16:41:10.075234 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:10.059066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50848 (local address 127.0.98.62:38541)
0619 16:41:10.059588 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:10.059604 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:10.059643 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:41:10.060470 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:10.060489 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:10.060875 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:10.061273 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:10.062447 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:10.063528 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:10.064717 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:10.065249 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:10.065460 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:10.068463 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:10.068470 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:10.068488 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:10.068522 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:10.071886 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:10.072479 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:10.072485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:10.072490 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:10.072616 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:10.072992 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:10.072996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:10.072998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:10.073441 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:10.073670 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:10.073975 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 16:41:10.074290 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":157,"threads_started":1}
I20260619 16:41:10.075341 2173 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e74dc107d0148a4a928ae1bce521b71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:41:10.075425 2173 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e74dc107d0148a4a928ae1bce521b71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:41:10.075553 2173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e74dc107d0148a4a928ae1bce521b71 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:10.076840 2173 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e74dc107d0148a4a928ae1bce521b71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e74dc107d0148a4a928ae1bce521b71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } }
I20260619 16:41:10.077257 2173 leader_election.cc:302] T 00000000000000000000000000000000 P 2e74dc107d0148a4a928ae1bce521b71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e74dc107d0148a4a928ae1bce521b71; no voters:
I20260619 16:41:10.077641 2173 leader_election.cc:288] T 00000000000000000000000000000000 P 2e74dc107d0148a4a928ae1bce521b71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:41:10.077759 2178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e74dc107d0148a4a928ae1bce521b71 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:10.077943 2178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e74dc107d0148a4a928ae1bce521b71 [term 1 LEADER]: Becoming Leader. State: Replica: 2e74dc107d0148a4a928ae1bce521b71, State: Running, Role: LEADER
I20260619 16:41:10.078327 2178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e74dc107d0148a4a928ae1bce521b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e74dc107d0148a4a928ae1bce521b71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } }
I20260619 16:41:10.079017 2173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e74dc107d0148a4a928ae1bce521b71 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:41:10.080204 2179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e74dc107d0148a4a928ae1bce521b71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e74dc107d0148a4a928ae1bce521b71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e74dc107d0148a4a928ae1bce521b71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } }
I20260619 16:41:10.080401 2179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e74dc107d0148a4a928ae1bce521b71 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:10.080813 2180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e74dc107d0148a4a928ae1bce521b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e74dc107d0148a4a928ae1bce521b71. Latest consensus state: current_term: 1 leader_uuid: "2e74dc107d0148a4a928ae1bce521b71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e74dc107d0148a4a928ae1bce521b71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } }
I20260619 16:41:10.080899 2187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:41:10.084111 2187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:41:10.084688 2180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e74dc107d0148a4a928ae1bce521b71 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:10.090109 2187 catalog_manager.cc:1347] Generated new cluster ID: 69c8a424cd5845a8ba02ec0adc68a92b
I20260619 16:41:10.090312 2187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:41:10.126449 2187 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:41:10.127565 2187 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:41:10.132598 2187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e74dc107d0148a4a928ae1bce521b71: Generated new TSK 0
I20260619 16:41:10.133586 2187 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:41:10.177340 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38541
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:10.310724 2199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:10.311064 2199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:10.311131 2199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:10.315594 2199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:10.315768 2199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:10.315824 2199 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 16:41:10.315879 2199 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:41:10.315948 2199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:41:10.323992 2199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:10.325578 2199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:10.326973 2199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:10.339625 2204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:10.339607 2207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:10.339856 2199 server_base.cc:1034] running on GCE node
W20260619 16:41:10.339607 2205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:10.340549 2199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:10.341472 2199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:10.342733 2199 hybrid_clock.cc:648] HybridClock initialized: now 1781887270342650 us; error 98 us; skew 500 ppm
Jun 19 16:41:10 dist-test-slave-03cd krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:10.346644 2199 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:41:10.348110 2199 webserver.cc:460] Webserver started at http://127.0.98.1:42967/ using document root <none> and password file <none>
I20260619 16:41:10.348897 2199 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:10.349007 2199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:10.349287 2199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:10.352008 2199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-0/data/instance:
uuid: "4ad5adf9d0e545eb8cccef1d682b37b9"
format_stamp: "Formatted at 2026-06-19 16:41:10 on dist-test-slave-03cd"
server_key: "b01f1ff6ddc81de9ae0e78674ec80992"
server_key_iv: "d0f7d89309c2ed32c92bbecc0e04150c"
server_key_version: "encryptionkey@0"
I20260619 16:41:10.352712 2199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4ad5adf9d0e545eb8cccef1d682b37b9"
format_stamp: "Formatted at 2026-06-19 16:41:10 on dist-test-slave-03cd"
server_key: "b01f1ff6ddc81de9ae0e78674ec80992"
server_key_iv: "d0f7d89309c2ed32c92bbecc0e04150c"
server_key_version: "encryptionkey@0"
I20260619 16:41:10.357803 2199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:41:10.361529 2214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:10.363420 2199 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:41:10.363754 2199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "4ad5adf9d0e545eb8cccef1d682b37b9"
format_stamp: "Formatted at 2026-06-19 16:41:10 on dist-test-slave-03cd"
server_key: "b01f1ff6ddc81de9ae0e78674ec80992"
server_key_iv: "d0f7d89309c2ed32c92bbecc0e04150c"
server_key_version: "encryptionkey@0"
I20260619 16:41:10.363966 2199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:10.385192 2199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:10.389639 2199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:10.389894 2199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:10.392033 2199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:10.392120 2199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:10.392204 2199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:10.392253 2199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:41:10 dist-test-slave-03cd krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:10.405143 2199 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46343
I20260619 16:41:10.406378 2329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46343 every 8 connection(s)
I20260619 16:41:10.406491 2199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:41:10.409991 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:10.393099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53209 (local address 127.0.98.62:38541)
0619 16:41:10.393321 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:10.393328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:10.394289 (+ 961us) server_negotiation.cc:407] Connection header received
0619 16:41:10.395438 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:10.395443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:10.395513 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:10.395577 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:10.397479 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:10.398523 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:10.400688 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:10.401204 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:10.401363 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:10.405527 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:10.405532 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:10.405536 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:10.405576 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:10.407387 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:10.408142 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:10.408147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:10.408148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:10.408275 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:10.408645 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:10.408647 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:10.408649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:10.408904 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:10.408996 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:10.409641 (+ 645us) server_negotiation.cc:299] Negotiation successful
0619 16:41:10.409794 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 16:41:10.410806 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:10.393534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.98.1:53209)
0619 16:41:10.394092 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:10.394146 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:10.395124 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:10.396144 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:10.396155 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:10.396597 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:10.397171 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:10.397220 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:10.398820 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:10.398826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:10.400502 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:10.400528 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:10.401458 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:10.401465 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:10.401806 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:10.402671 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:10.402754 (+ 83us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:10.405241 (+ 2487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:10.407551 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:10.407558 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:10.407577 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:10.407989 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:10.408389 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:10.408392 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:10.408393 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:10.408534 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:10.409049 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:10.409055 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:10.409345 (+ 290us) client_negotiation.cc:769] Sending connection context
0619 16:41:10.409539 (+ 194us) client_negotiation.cc:240] Negotiation successful
0619 16:41:10.409825 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":165,"threads_started":1}
I20260619 16:41:10.412765 2330 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38541
I20260619 16:41:10.413007 2330 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:10.413749 2330 heartbeater.cc:511] Master 127.0.98.62:38541 requested a full tablet report, sending...
I20260619 16:41:10.415823 2127 ts_manager.cc:194] Registered new tserver with Master: 4ad5adf9d0e545eb8cccef1d682b37b9 (127.0.98.1:46343)
I20260619 16:41:10.416332 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 2199
I20260619 16:41:10.416471 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:41:10.416813 392 external_mini_cluster.cc:1428] Setting key 9a3535dcf7e237c38424524d64e223b8
I20260619 16:41:10.417621 2127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53209
I20260619 16:41:10.425585 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:10.415485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56525 (local address 127.0.98.62:38541)
0619 16:41:10.415637 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:10.415643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:10.415918 (+ 275us) server_negotiation.cc:407] Connection header received
0619 16:41:10.416056 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:10.416059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:10.416114 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:10.416218 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:10.416829 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:10.417637 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:10.419405 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:10.419770 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:10.419886 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:10.421444 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:10.421451 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:10.421455 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:10.421490 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:10.423344 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:10.424064 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:10.424067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:10.424068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:10.424153 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:10.424503 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:10.424505 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:10.424506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:10.424758 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:10.424841 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:10.425262 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 16:41:10.425391 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:41:10.425746 2336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:10.415323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.98.1:56525)
0619 16:41:10.415815 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:10.415829 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:10.415935 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:10.416272 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:10.416275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:10.416521 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:10.416687 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:10.416696 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:10.417795 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:10.417798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:10.419279 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:10.419290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:10.419915 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:10.419920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:10.420061 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:10.420787 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:10.420808 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:10.421275 (+ 467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:10.423506 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:10.423513 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:10.423519 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:10.423920 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:10.424268 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:10.424272 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:10.424274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:10.424393 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:10.424883 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:10.424888 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:10.425054 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 16:41:10.425203 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 16:41:10.425425 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":126,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:41:10.452324 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38541
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:10.576875 2339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:10.577246 2339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:10.577332 2339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:10.582319 2339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:10.582415 2339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:10.582489 2339 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 16:41:10.582533 2339 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:41:10.582595 2339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:41:10.588850 2339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:10.590071 2339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:10.591193 2339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:10.602882 2347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:10.602909 2345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:10.602882 2344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:10.603422 2339 server_base.cc:1034] running on GCE node
I20260619 16:41:10.604086 2339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:10.604975 2339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:10.606209 2339 hybrid_clock.cc:648] HybridClock initialized: now 1781887270606183 us; error 50 us; skew 500 ppm
Jun 19 16:41:10 dist-test-slave-03cd krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:10.610028 2339 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:41:10.611300 2339 webserver.cc:460] Webserver started at http://127.0.98.2:42635/ using document root <none> and password file <none>
I20260619 16:41:10.612118 2339 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:10.612226 2339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:10.612586 2339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:10.615319 2339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-1/data/instance:
uuid: "05e53be59c9f40c1806e9662993fd161"
format_stamp: "Formatted at 2026-06-19 16:41:10 on dist-test-slave-03cd"
server_key: "9babb87b3522d079c6dd6b40a52f1554"
server_key_iv: "19fff674e859378a5835b1c3c7c9a49a"
server_key_version: "encryptionkey@0"
I20260619 16:41:10.616096 2339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-1/wal/instance:
uuid: "05e53be59c9f40c1806e9662993fd161"
format_stamp: "Formatted at 2026-06-19 16:41:10 on dist-test-slave-03cd"
server_key: "9babb87b3522d079c6dd6b40a52f1554"
server_key_iv: "19fff674e859378a5835b1c3c7c9a49a"
server_key_version: "encryptionkey@0"
I20260619 16:41:10.621395 2339 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 16:41:10.625082 2354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:10.627038 2339 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:41:10.627285 2339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "05e53be59c9f40c1806e9662993fd161"
format_stamp: "Formatted at 2026-06-19 16:41:10 on dist-test-slave-03cd"
server_key: "9babb87b3522d079c6dd6b40a52f1554"
server_key_iv: "19fff674e859378a5835b1c3c7c9a49a"
server_key_version: "encryptionkey@0"
I20260619 16:41:10.627491 2339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:10.647289 2339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:10.651492 2339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:10.651775 2339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:10.653586 2339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:10.653672 2339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:10.653761 2339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:10.653810 2339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:41:10 dist-test-slave-03cd krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:10.667750 2339 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40965
I20260619 16:41:10.667816 2469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40965 every 8 connection(s)
I20260619 16:41:10.669368 2339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:41:10.671100 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 2339
I20260619 16:41:10.671160 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:10.655123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44803 (local address 127.0.98.62:38541)
0619 16:41:10.655287 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:10.655291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:10.656855 (+ 1564us) server_negotiation.cc:407] Connection header received
0619 16:41:10.658253 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:10.658257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:10.658320 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:10.658412 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:10.660240 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:10.660895 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:10.662587 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:10.662817 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:10.662958 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:10.666048 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:10.666052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:10.666054 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:10.666082 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:10.668084 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:10.668729 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:10.668736 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:10.668740 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:10.668857 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:10.669306 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:10.669313 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:10.669318 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:10.669650 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:10.669775 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:10.670697 (+ 922us) server_negotiation.cc:299] Negotiation successful
0619 16:41:10.670953 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:41:10.671226 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:41:10.671556 392 external_mini_cluster.cc:1428] Setting key b18192511f08fa53ecf7416a8f053f7e
I20260619 16:41:10.672209 2363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:10.655874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.98.2:44803)
0619 16:41:10.656620 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:10.656689 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:10.657974 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:10.658689 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:10.658698 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:10.659120 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:10.660032 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:10.660052 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:10.661111 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:10.661117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:10.662450 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:10.662467 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:10.663029 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:10.663033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:10.663246 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:10.664039 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:10.664059 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:10.665812 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:10.668262 (+ 2450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:10.668268 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:10.668282 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:10.668579 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:10.668964 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:10.668966 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:10.668967 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:10.669180 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:10.669861 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:10.669867 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:10.670319 (+ 452us) client_negotiation.cc:769] Sending connection context
0619 16:41:10.670640 (+ 321us) client_negotiation.cc:240] Negotiation successful
0619 16:41:10.671015 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":159,"threads_started":1}
I20260619 16:41:10.673708 2470 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38541
I20260619 16:41:10.673918 2470 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:10.674723 2470 heartbeater.cc:511] Master 127.0.98.62:38541 requested a full tablet report, sending...
I20260619 16:41:10.676265 2126 ts_manager.cc:194] Registered new tserver with Master: 05e53be59c9f40c1806e9662993fd161 (127.0.98.2:40965)
I20260619 16:41:10.677263 2126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44803
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:41:10.687179 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:10.676302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41577 (local address 127.0.98.62:38541)
0619 16:41:10.676445 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:10.676450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:10.676710 (+ 260us) server_negotiation.cc:407] Connection header received
0619 16:41:10.676880 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:10.676883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:10.676937 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:10.677034 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:10.677706 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:10.678589 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:10.679885 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:10.680335 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:10.680509 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:10.682100 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:10.682206 (+ 106us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:10.682218 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:10.682269 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:10.684444 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:10.685143 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:10.685146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:10.685147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:10.685239 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:10.685593 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:10.685600 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:10.685605 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:10.686150 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:10.686306 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:10.686804 (+ 498us) server_negotiation.cc:299] Negotiation successful
0619 16:41:10.686983 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:41:10.687183 2477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:10.676242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.98.2:41577)
0619 16:41:10.676615 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:10.676628 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:10.676766 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:10.677051 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:10.677054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:10.677304 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:10.677475 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:10.677490 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:10.678789 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:10.678793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:10.679639 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:10.679651 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:10.680507 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:10.680510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:10.680592 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:10.681475 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:10.681487 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:10.681902 (+ 415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:10.684611 (+ 2709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:10.684614 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:10.684615 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:10.684918 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:10.685346 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:10.685349 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:10.685350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:10.685492 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:10.686390 (+ 898us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:10.686393 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:10.686569 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 16:41:10.686688 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 16:41:10.687006 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":104,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:41:10.713268 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38541
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:10.862579 2479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:10.862963 2479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:10.863021 2479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:10.867563 2479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:10.867692 2479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:10.867743 2479 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 16:41:10.867758 2479 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:41:10.867856 2479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:41:10.874611 2479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:10.876156 2479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:10.877475 2479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:41:10.890697 2479 server_base.cc:1034] running on GCE node
W20260619 16:41:10.890543 2484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:10.890597 2485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:10.890583 2487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:10.891639 2479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:10.892951 2479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:10.894207 2479 hybrid_clock.cc:648] HybridClock initialized: now 1781887270894166 us; error 67 us; skew 500 ppm
Jun 19 16:41:10 dist-test-slave-03cd krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:10.899201 2479 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:41:10.900751 2479 webserver.cc:460] Webserver started at http://127.0.98.3:41927/ using document root <none> and password file <none>
I20260619 16:41:10.901757 2479 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:10.901863 2479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:10.902215 2479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:10.905259 2479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-2/data/instance:
uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9"
format_stamp: "Formatted at 2026-06-19 16:41:10 on dist-test-slave-03cd"
server_key: "6c0b2a6a94960dbffaee61bcff32fae1"
server_key_iv: "18af6997abc2887a189eb2db473631f2"
server_key_version: "encryptionkey@0"
I20260619 16:41:10.906098 2479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9"
format_stamp: "Formatted at 2026-06-19 16:41:10 on dist-test-slave-03cd"
server_key: "6c0b2a6a94960dbffaee61bcff32fae1"
server_key_iv: "18af6997abc2887a189eb2db473631f2"
server_key_version: "encryptionkey@0"
I20260619 16:41:10.912992 2479 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260619 16:41:10.916857 2494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:10.918812 2479 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260619 16:41:10.919008 2479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9"
format_stamp: "Formatted at 2026-06-19 16:41:10 on dist-test-slave-03cd"
server_key: "6c0b2a6a94960dbffaee61bcff32fae1"
server_key_iv: "18af6997abc2887a189eb2db473631f2"
server_key_version: "encryptionkey@0"
I20260619 16:41:10.919164 2479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:10.950381 2479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:10.954581 2479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:10.955018 2479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:10.956874 2479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:10.957000 2479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:10.957069 2479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:10.957124 2479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:10.970988 2479 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45745
I20260619 16:41:10.971127 2609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45745 every 8 connection(s)
Jun 19 16:41:10 dist-test-slave-03cd krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:10.973557 2479 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:41:10.974000 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 2479
I20260619 16:41:10.974129 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:41:10.974475 392 external_mini_cluster.cc:1428] Setting key 46210040bebc2795d0c44b96d518d0cb
I20260619 16:41:10.982777 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:10.959097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60337 (local address 127.0.98.62:38541)
0619 16:41:10.959420 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:10.959425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:10.961010 (+ 1585us) server_negotiation.cc:407] Connection header received
0619 16:41:10.962616 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:10.962621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:10.962702 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:10.962802 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:10.965191 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:10.966188 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:10.968750 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:10.969217 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:10.969367 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:10.974975 (+ 5608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:10.974980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:10.974984 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:10.975030 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:10.978217 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:10.979348 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:10.979352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:10.979354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:10.979634 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:10.980518 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:10.980521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:10.980523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:10.981009 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:10.981129 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:10.982312 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0619 16:41:10.982609 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260619 16:41:10.983592 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:10.959531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.98.3:60337)
0619 16:41:10.960716 (+ 1185us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:10.960788 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:10.962237 (+ 1449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:10.963140 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:10.963156 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:10.963969 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:10.964967 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:10.964997 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:10.966453 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:10.966461 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:10.968469 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:10.968491 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:10.969717 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:10.969724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:10.969991 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:10.971125 (+ 1134us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:10.971157 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:10.974534 (+ 3377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:10.978658 (+ 4124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:10.978670 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:10.978700 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:10.979175 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:10.979877 (+ 702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:10.979883 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:10.979886 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:10.980197 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:10.981212 (+ 1015us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:10.981220 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:10.981918 (+ 698us) client_negotiation.cc:769] Sending connection context
0619 16:41:10.982172 (+ 254us) client_negotiation.cc:240] Negotiation successful
0619 16:41:10.982535 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"thread_start_us":192,"threads_started":1}
I20260619 16:41:10.985634 2610 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38541
I20260619 16:41:10.985946 2610 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:10.986706 2610 heartbeater.cc:511] Master 127.0.98.62:38541 requested a full tablet report, sending...
I20260619 16:41:10.988700 2127 ts_manager.cc:194] Registered new tserver with Master: 3d782f18c34e43a3b39fc58ffdc8e1d9 (127.0.98.3:45745)
I20260619 16:41:10.989946 2127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60337
I20260619 16:41:10.991848 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:41:11.000137 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:10.989389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.98.3:38859)
0619 16:41:10.989913 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:10.989934 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:10.990159 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:10.990570 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:10.990573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:10.990874 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:10.991072 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:10.991080 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:10.992409 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:10.992413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:10.993086 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:10.993097 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:10.993640 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:10.993644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:10.993748 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:10.994511 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:10.994532 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:10.995160 (+ 628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:10.997706 (+ 2546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:10.997714 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:10.997719 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:10.998116 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:10.998605 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:10.998612 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:10.998615 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:10.998746 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:10.999350 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:10.999353 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:10.999501 (+ 148us) client_negotiation.cc:769] Sending connection context
0619 16:41:10.999620 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 16:41:10.999939 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":1,"thread_start_us":222,"threads_started":1}
I20260619 16:41:11.000223 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:10.989755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38859 (local address 127.0.98.62:38541)
0619 16:41:10.989888 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:10.989893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:10.990061 (+ 168us) server_negotiation.cc:407] Connection header received
0619 16:41:10.990284 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:10.990287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:10.990371 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:10.990474 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:10.991228 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:10.992237 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:10.993240 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:10.993490 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:10.993624 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:10.995451 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:10.995455 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:10.995458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:10.995499 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:10.997514 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:10.998284 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:10.998287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:10.998289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:10.998437 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:10.998874 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:10.998881 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:10.998886 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:10.999199 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:10.999333 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:10.999759 (+ 426us) server_negotiation.cc:299] Negotiation successful
0619 16:41:10.999950 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Jun 19 16:41:11 dist-test-slave-03cd krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:41:11 dist-test-slave-03cd krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:11.027261 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:11.016690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50858 (local address 127.0.98.62:38541)
0619 16:41:11.016841 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:11.016845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:11.016862 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:41:11.016921 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:11.016924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:11.016971 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:11.017077 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:11.017992 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.018579 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.019744 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.020065 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.020170 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:11.023348 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:11.023353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:11.023355 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:11.023392 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:11.025314 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:11.025877 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:11.025880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:11.025881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:11.025982 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:11.026323 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:11.026326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:11.026327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:11.026554 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:11.026684 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:11.026986 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 16:41:11.027111 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:41:11.032269 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:41:11.036751 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:41:11.057412 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:11.049912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45745 (local address 127.0.0.1:44770)
0619 16:41:11.050281 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:11.050295 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:11.050384 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:11.051718 (+ 1334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:11.051726 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:11.051755 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:11.052149 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:11.052164 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.053819 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.053823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:11.055346 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:11.055359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.056599 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.056602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:11.056879 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:11.056932 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 16:41:11.057039 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 16:41:11.057242 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":101,"thread_start_us":122,"threads_started":1}
I20260619 16:41:11.058261 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:11.050261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40965 (local address 127.0.0.1:34622)
0619 16:41:11.050590 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:11.050604 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:11.050689 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:11.052069 (+ 1380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:11.052073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:11.052091 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:11.052279 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:11.052286 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.054772 (+ 2486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.054776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:11.056142 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:11.056151 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.057676 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.057680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:11.057788 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:11.057818 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 16:41:11.057904 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 16:41:11.058014 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":66,"thread_start_us":138,"threads_started":1}
I20260619 16:41:11.058781 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:11.050671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44770 (local address 127.0.98.3:45745)
0619 16:41:11.051239 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:11.051250 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:11.051270 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:41:11.051343 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:11.051347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:11.051504 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:11.051681 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:11.052332 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.053632 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.055524 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.056442 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.058026 (+ 1584us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:11.058239 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:11.058297 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 16:41:11.058618 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":152,"threads_started":1}
I20260619 16:41:11.059345 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:11.050784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34622 (local address 127.0.98.2:40965)
0619 16:41:11.051297 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:11.051309 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:11.051334 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:41:11.051414 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:11.051420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:11.051692 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:11.051852 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:11.052441 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.054467 (+ 2026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.056297 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.057211 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.057319 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:11.058590 (+ 1271us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:11.058672 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 16:41:11.058815 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":127,"threads_started":1}
I20260619 16:41:11.059700 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:11.049292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55712 (local address 127.0.98.1:46343)
0619 16:41:11.049815 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:11.049824 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:11.050266 (+ 442us) server_negotiation.cc:407] Connection header received
0619 16:41:11.050367 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:11.050375 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:11.050690 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:11.050892 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:11.052285 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.053474 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.056048 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.057124 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.057237 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:11.057476 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:11.059362 (+ 1886us) server_negotiation.cc:299] Negotiation successful
0619 16:41:11.059536 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":160,"threads_started":1}
I20260619 16:41:11.060163 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:11.049024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46343 (local address 127.0.0.1:55712)
0619 16:41:11.049785 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:11.049870 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:11.050082 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:11.051893 (+ 1811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:11.051896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:11.051910 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:11.052124 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:11.052133 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.053623 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.053627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:11.055862 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:11.055877 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.059041 (+ 3164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.059045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:11.059174 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:11.059200 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 16:41:11.059923 (+ 723us) client_negotiation.cc:240] Negotiation successful
0619 16:41:11.060028 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":455,"thread_start_us":103,"threads_started":1}
I20260619 16:41:11.062438 2544 tablet_service.cc:1467] Processing CreateTablet for tablet d4fbebc2ae1b494f8f57116e63e6a55a (DEFAULT_TABLE table=test-table [id=503fc2ac80fc4822bb0eb44e31c0543f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:11.063225 2404 tablet_service.cc:1467] Processing CreateTablet for tablet d4fbebc2ae1b494f8f57116e63e6a55a (DEFAULT_TABLE table=test-table [id=503fc2ac80fc4822bb0eb44e31c0543f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:11.064046 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4fbebc2ae1b494f8f57116e63e6a55a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:11.064728 2404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4fbebc2ae1b494f8f57116e63e6a55a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:11.064250 2264 tablet_service.cc:1467] Processing CreateTablet for tablet d4fbebc2ae1b494f8f57116e63e6a55a (DEFAULT_TABLE table=test-table [id=503fc2ac80fc4822bb0eb44e31c0543f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:11.065936 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4fbebc2ae1b494f8f57116e63e6a55a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:11.075480 2633 tablet_bootstrap.cc:492] T d4fbebc2ae1b494f8f57116e63e6a55a P 4ad5adf9d0e545eb8cccef1d682b37b9: Bootstrap starting.
I20260619 16:41:11.076272 2635 tablet_bootstrap.cc:492] T d4fbebc2ae1b494f8f57116e63e6a55a P 05e53be59c9f40c1806e9662993fd161: Bootstrap starting.
I20260619 16:41:11.076272 2634 tablet_bootstrap.cc:492] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9: Bootstrap starting.
I20260619 16:41:11.078156 2633 tablet_bootstrap.cc:654] T d4fbebc2ae1b494f8f57116e63e6a55a P 4ad5adf9d0e545eb8cccef1d682b37b9: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:11.078994 2634 tablet_bootstrap.cc:654] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:11.078994 2635 tablet_bootstrap.cc:654] T d4fbebc2ae1b494f8f57116e63e6a55a P 05e53be59c9f40c1806e9662993fd161: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:11.079310 2633 log.cc:824] T d4fbebc2ae1b494f8f57116e63e6a55a P 4ad5adf9d0e545eb8cccef1d682b37b9: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:11.080082 2634 log.cc:824] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:11.080082 2635 log.cc:824] T d4fbebc2ae1b494f8f57116e63e6a55a P 05e53be59c9f40c1806e9662993fd161: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:11.082253 2633 tablet_bootstrap.cc:492] T d4fbebc2ae1b494f8f57116e63e6a55a P 4ad5adf9d0e545eb8cccef1d682b37b9: No bootstrap required, opened a new log
I20260619 16:41:11.082504 2635 tablet_bootstrap.cc:492] T d4fbebc2ae1b494f8f57116e63e6a55a P 05e53be59c9f40c1806e9662993fd161: No bootstrap required, opened a new log
I20260619 16:41:11.082504 2634 tablet_bootstrap.cc:492] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9: No bootstrap required, opened a new log
I20260619 16:41:11.082583 2633 ts_tablet_manager.cc:1397] T d4fbebc2ae1b494f8f57116e63e6a55a P 4ad5adf9d0e545eb8cccef1d682b37b9: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260619 16:41:11.082728 2635 ts_tablet_manager.cc:1397] T d4fbebc2ae1b494f8f57116e63e6a55a P 05e53be59c9f40c1806e9662993fd161: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 16:41:11.082728 2634 ts_tablet_manager.cc:1397] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 16:41:11.087481 2633 raft_consensus.cc:348] T d4fbebc2ae1b494f8f57116e63e6a55a P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } }
I20260619 16:41:11.087494 2635 raft_consensus.cc:348] T d4fbebc2ae1b494f8f57116e63e6a55a P 05e53be59c9f40c1806e9662993fd161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } }
I20260619 16:41:11.087494 2634 raft_consensus.cc:348] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } }
I20260619 16:41:11.087890 2635 raft_consensus.cc:374] T d4fbebc2ae1b494f8f57116e63e6a55a P 05e53be59c9f40c1806e9662993fd161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:11.087890 2634 raft_consensus.cc:374] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:11.087895 2633 raft_consensus.cc:374] T d4fbebc2ae1b494f8f57116e63e6a55a P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:11.087958 2635 raft_consensus.cc:729] T d4fbebc2ae1b494f8f57116e63e6a55a P 05e53be59c9f40c1806e9662993fd161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05e53be59c9f40c1806e9662993fd161, State: Initialized, Role: FOLLOWER
I20260619 16:41:11.087970 2634 raft_consensus.cc:729] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d782f18c34e43a3b39fc58ffdc8e1d9, State: Initialized, Role: FOLLOWER
I20260619 16:41:11.087973 2633 raft_consensus.cc:729] T d4fbebc2ae1b494f8f57116e63e6a55a P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ad5adf9d0e545eb8cccef1d682b37b9, State: Initialized, Role: FOLLOWER
I20260619 16:41:11.088562 2633 consensus_queue.cc:260] T d4fbebc2ae1b494f8f57116e63e6a55a P 4ad5adf9d0e545eb8cccef1d682b37b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } }
I20260619 16:41:11.088574 2635 consensus_queue.cc:260] T d4fbebc2ae1b494f8f57116e63e6a55a P 05e53be59c9f40c1806e9662993fd161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } }
I20260619 16:41:11.088574 2634 consensus_queue.cc:260] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } }
I20260619 16:41:11.089581 2330 heartbeater.cc:503] Master 127.0.98.62:38541 was elected leader, sending a full tablet report...
I20260619 16:41:11.089972 2635 ts_tablet_manager.cc:1428] T d4fbebc2ae1b494f8f57116e63e6a55a P 05e53be59c9f40c1806e9662993fd161: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260619 16:41:11.090026 2634 ts_tablet_manager.cc:1428] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20260619 16:41:11.090054 2633 ts_tablet_manager.cc:1428] T d4fbebc2ae1b494f8f57116e63e6a55a P 4ad5adf9d0e545eb8cccef1d682b37b9: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 16:41:11.090500 2470 heartbeater.cc:503] Master 127.0.98.62:38541 was elected leader, sending a full tablet report...
I20260619 16:41:11.090569 2610 heartbeater.cc:503] Master 127.0.98.62:38541 was elected leader, sending a full tablet report...
I20260619 16:41:11.125029 2641 raft_consensus.cc:482] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:41:11.125313 2641 raft_consensus.cc:504] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } }
I20260619 16:41:11.126953 2641 leader_election.cc:288] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ad5adf9d0e545eb8cccef1d682b37b9 (127.0.98.1:46343), 05e53be59c9f40c1806e9662993fd161 (127.0.98.2:40965)
I20260619 16:41:11.132449 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:11.127627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44665 (local address 127.0.98.1:46343)
0619 16:41:11.127819 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:11.127824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:11.128483 (+ 659us) server_negotiation.cc:407] Connection header received
0619 16:41:11.128670 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:11.128673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:11.128739 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:11.128834 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:11.129234 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.129759 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.130906 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.131473 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.131604 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:11.131705 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:11.132204 (+ 499us) server_negotiation.cc:299] Negotiation successful
0619 16:41:11.132292 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 16:41:11.132472 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:11.127592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46343 (local address 127.0.98.3:44665)
0619 16:41:11.128323 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:11.128340 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:11.128548 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:11.128844 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:11.128856 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:11.128872 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:11.129049 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:11.129071 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.129880 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.129886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:11.130764 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:11.130773 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.131625 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.131629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:11.131985 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:11.132015 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 16:41:11.132185 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 16:41:11.132283 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":320,"thread_start_us":149,"threads_started":1}
I20260619 16:41:11.132575 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:11.127592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40965 (local address 127.0.98.3:58137)
0619 16:41:11.127829 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:11.127848 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:11.128028 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:11.128471 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:11.128580 (+ 109us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:11.128607 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:11.128763 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:11.128771 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.129636 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.129642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:11.130975 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:11.130996 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.132043 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.132047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:11.132156 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:11.132180 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 16:41:11.132257 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 16:41:11.132368 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142}
I20260619 16:41:11.133093 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:11.127852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58137 (local address 127.0.98.2:40965)
0619 16:41:11.128005 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:11.128010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:11.128029 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:41:11.128281 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:11.128285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:11.128340 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:11.128458 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:11.128901 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.129496 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.131179 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.131792 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.131896 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:11.131974 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:11.132801 (+ 827us) server_negotiation.cc:299] Negotiation successful
0619 16:41:11.132932 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:41:11.133538 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fbebc2ae1b494f8f57116e63e6a55a" candidate_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" is_pre_election: true
I20260619 16:41:11.133816 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fbebc2ae1b494f8f57116e63e6a55a" candidate_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05e53be59c9f40c1806e9662993fd161" is_pre_election: true
I20260619 16:41:11.134013 2284 raft_consensus.cc:2454] T d4fbebc2ae1b494f8f57116e63e6a55a P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d782f18c34e43a3b39fc58ffdc8e1d9 in term 0.
I20260619 16:41:11.134053 2424 raft_consensus.cc:2454] T d4fbebc2ae1b494f8f57116e63e6a55a P 05e53be59c9f40c1806e9662993fd161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d782f18c34e43a3b39fc58ffdc8e1d9 in term 0.
I20260619 16:41:11.134725 2496 leader_election.cc:302] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 05e53be59c9f40c1806e9662993fd161, 3d782f18c34e43a3b39fc58ffdc8e1d9, 4ad5adf9d0e545eb8cccef1d682b37b9; no voters:
I20260619 16:41:11.135103 2641 raft_consensus.cc:2781] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:41:11.135201 2641 raft_consensus.cc:482] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:41:11.135246 2641 raft_consensus.cc:3037] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:11.136780 2641 raft_consensus.cc:504] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } }
I20260619 16:41:11.137257 2641 leader_election.cc:288] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4ad5adf9d0e545eb8cccef1d682b37b9 (127.0.98.1:46343), 05e53be59c9f40c1806e9662993fd161 (127.0.98.2:40965)
I20260619 16:41:11.137863 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fbebc2ae1b494f8f57116e63e6a55a" candidate_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05e53be59c9f40c1806e9662993fd161"
I20260619 16:41:11.138072 2424 raft_consensus.cc:3037] T d4fbebc2ae1b494f8f57116e63e6a55a P 05e53be59c9f40c1806e9662993fd161 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:11.139493 2424 raft_consensus.cc:2454] T d4fbebc2ae1b494f8f57116e63e6a55a P 05e53be59c9f40c1806e9662993fd161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d782f18c34e43a3b39fc58ffdc8e1d9 in term 1.
I20260619 16:41:11.140103 2496 leader_election.cc:302] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05e53be59c9f40c1806e9662993fd161, 3d782f18c34e43a3b39fc58ffdc8e1d9; no voters:
I20260619 16:41:11.140144 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fbebc2ae1b494f8f57116e63e6a55a" candidate_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9"
I20260619 16:41:11.140311 2284 raft_consensus.cc:3037] T d4fbebc2ae1b494f8f57116e63e6a55a P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:11.140537 2641 raft_consensus.cc:2781] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:11.140882 2641 raft_consensus.cc:686] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 1 LEADER]: Becoming Leader. State: Replica: 3d782f18c34e43a3b39fc58ffdc8e1d9, State: Running, Role: LEADER
I20260619 16:41:11.141404 2641 consensus_queue.cc:237] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } }
I20260619 16:41:11.142091 2284 raft_consensus.cc:2454] T d4fbebc2ae1b494f8f57116e63e6a55a P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d782f18c34e43a3b39fc58ffdc8e1d9 in term 1.
I20260619 16:41:11.145488 2127 catalog_manager.cc:5495] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d782f18c34e43a3b39fc58ffdc8e1d9 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } health_report { overall_health: UNKNOWN } } }
W20260619 16:41:11.158390 2331 tablet.cc:2369] T d4fbebc2ae1b494f8f57116e63e6a55a P 4ad5adf9d0e545eb8cccef1d682b37b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:41:11.171131 2471 tablet.cc:2369] T d4fbebc2ae1b494f8f57116e63e6a55a P 05e53be59c9f40c1806e9662993fd161: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 16:41:11 dist-test-slave-03cd krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:11.192492 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:11.178114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.0.1:50868)
0619 16:41:11.178731 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:11.178747 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:11.178941 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:11.179841 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:11.179845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:11.180631 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:11.180783 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:11.180792 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.181877 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.181880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:11.182553 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:11.182564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.183989 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.183995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:11.184096 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:11.185150 (+ 1054us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:11.185180 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:11.187838 (+ 2658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:11.190354 (+ 2516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:11.190371 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:11.190380 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:11.190656 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:11.190963 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:11.190966 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:11.190968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:11.191049 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:11.191641 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:11.191651 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:11.191838 (+ 187us) client_negotiation.cc:769] Sending connection context
0619 16:41:11.191983 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 16:41:11.192333 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":267,"threads_started":1}
I20260619 16:41:11.192682 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:11.178342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50868 (local address 127.0.98.62:38541)
0619 16:41:11.178605 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:11.178611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:11.178882 (+ 271us) server_negotiation.cc:407] Connection header received
0619 16:41:11.179441 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:11.179447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:11.179508 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:11.179758 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:11.181027 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.181671 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.182996 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.183343 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.183481 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:11.188053 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:11.188061 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:11.188063 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:11.188088 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:11.190181 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:11.190768 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:11.190774 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:11.190777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:11.190859 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:11.191154 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:11.191157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:11.191159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:11.191394 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:11.191637 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:11.192295 (+ 658us) server_negotiation.cc:299] Negotiation successful
0619 16:41:11.192517 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260619 16:41:11.200134 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50868:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 16:41:11.201215 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:41:11.208990 2264 tablet_service.cc:1467] Processing CreateTablet for tablet ba77e66f2b894e1dad718bc5db89acef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5af8e3053e3d4b3d98ebc1a5a860ade8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:41:11.209232 2544 tablet_service.cc:1467] Processing CreateTablet for tablet ba77e66f2b894e1dad718bc5db89acef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5af8e3053e3d4b3d98ebc1a5a860ade8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:41:11.209252 2404 tablet_service.cc:1467] Processing CreateTablet for tablet ba77e66f2b894e1dad718bc5db89acef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5af8e3053e3d4b3d98ebc1a5a860ade8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:41:11.209934 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba77e66f2b894e1dad718bc5db89acef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:11.209935 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba77e66f2b894e1dad718bc5db89acef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:11.210291 2404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba77e66f2b894e1dad718bc5db89acef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:11.213608 2633 tablet_bootstrap.cc:492] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9: Bootstrap starting.
I20260619 16:41:11.213603 2634 tablet_bootstrap.cc:492] T ba77e66f2b894e1dad718bc5db89acef P 3d782f18c34e43a3b39fc58ffdc8e1d9: Bootstrap starting.
I20260619 16:41:11.215122 2633 tablet_bootstrap.cc:654] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:11.215123 2634 tablet_bootstrap.cc:654] T ba77e66f2b894e1dad718bc5db89acef P 3d782f18c34e43a3b39fc58ffdc8e1d9: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:11.215804 2635 tablet_bootstrap.cc:492] T ba77e66f2b894e1dad718bc5db89acef P 05e53be59c9f40c1806e9662993fd161: Bootstrap starting.
I20260619 16:41:11.217002 2635 tablet_bootstrap.cc:654] T ba77e66f2b894e1dad718bc5db89acef P 05e53be59c9f40c1806e9662993fd161: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:11.217238 2633 tablet_bootstrap.cc:492] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9: No bootstrap required, opened a new log
I20260619 16:41:11.217238 2634 tablet_bootstrap.cc:492] T ba77e66f2b894e1dad718bc5db89acef P 3d782f18c34e43a3b39fc58ffdc8e1d9: No bootstrap required, opened a new log
I20260619 16:41:11.217371 2634 ts_tablet_manager.cc:1397] T ba77e66f2b894e1dad718bc5db89acef P 3d782f18c34e43a3b39fc58ffdc8e1d9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 16:41:11.217371 2633 ts_tablet_manager.cc:1397] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260619 16:41:11.217995 2634 raft_consensus.cc:348] T ba77e66f2b894e1dad718bc5db89acef P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:11.217995 2633 raft_consensus.cc:348] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:11.218159 2633 raft_consensus.cc:374] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:11.218160 2634 raft_consensus.cc:374] T ba77e66f2b894e1dad718bc5db89acef P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:11.218207 2634 raft_consensus.cc:729] T ba77e66f2b894e1dad718bc5db89acef P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d782f18c34e43a3b39fc58ffdc8e1d9, State: Initialized, Role: FOLLOWER
I20260619 16:41:11.218207 2633 raft_consensus.cc:729] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ad5adf9d0e545eb8cccef1d682b37b9, State: Initialized, Role: FOLLOWER
I20260619 16:41:11.218374 2634 consensus_queue.cc:260] T ba77e66f2b894e1dad718bc5db89acef P 3d782f18c34e43a3b39fc58ffdc8e1d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:11.218429 2633 consensus_queue.cc:260] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:11.218665 2635 tablet_bootstrap.cc:492] T ba77e66f2b894e1dad718bc5db89acef P 05e53be59c9f40c1806e9662993fd161: No bootstrap required, opened a new log
I20260619 16:41:11.218711 2634 ts_tablet_manager.cc:1428] T ba77e66f2b894e1dad718bc5db89acef P 3d782f18c34e43a3b39fc58ffdc8e1d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:41:11.218716 2633 ts_tablet_manager.cc:1428] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 16:41:11.218791 2635 ts_tablet_manager.cc:1397] T ba77e66f2b894e1dad718bc5db89acef P 05e53be59c9f40c1806e9662993fd161: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:41:11.218830 2639 tablet_replica.cc:439] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } } }
I20260619 16:41:11.219072 2639 tablet_replica.cc:442] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:41:11.219341 2641 tablet_replica.cc:439] T ba77e66f2b894e1dad718bc5db89acef P 3d782f18c34e43a3b39fc58ffdc8e1d9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } } }
I20260619 16:41:11.219501 2641 tablet_replica.cc:442] T ba77e66f2b894e1dad718bc5db89acef P 3d782f18c34e43a3b39fc58ffdc8e1d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:41:11.219740 2635 raft_consensus.cc:348] T ba77e66f2b894e1dad718bc5db89acef P 05e53be59c9f40c1806e9662993fd161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:11.219868 2635 raft_consensus.cc:374] T ba77e66f2b894e1dad718bc5db89acef P 05e53be59c9f40c1806e9662993fd161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:11.219911 2635 raft_consensus.cc:729] T ba77e66f2b894e1dad718bc5db89acef P 05e53be59c9f40c1806e9662993fd161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05e53be59c9f40c1806e9662993fd161, State: Initialized, Role: FOLLOWER
I20260619 16:41:11.220052 2635 consensus_queue.cc:260] T ba77e66f2b894e1dad718bc5db89acef P 05e53be59c9f40c1806e9662993fd161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:11.220320 2640 tablet_replica.cc:439] T ba77e66f2b894e1dad718bc5db89acef P 05e53be59c9f40c1806e9662993fd161: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } } }
I20260619 16:41:11.220454 2640 tablet_replica.cc:442] T ba77e66f2b894e1dad718bc5db89acef P 05e53be59c9f40c1806e9662993fd161: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:41:11.220888 2635 ts_tablet_manager.cc:1428] T ba77e66f2b894e1dad718bc5db89acef P 05e53be59c9f40c1806e9662993fd161: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20260619 16:41:11.225862 2611 tablet.cc:2369] T ba77e66f2b894e1dad718bc5db89acef P 3d782f18c34e43a3b39fc58ffdc8e1d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:41:11.327522 2639 raft_consensus.cc:482] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:41:11.327747 2639 raft_consensus.cc:504] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:11.329097 2639 leader_election.cc:288] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05e53be59c9f40c1806e9662993fd161 (127.0.98.2:40965), 3d782f18c34e43a3b39fc58ffdc8e1d9 (127.0.98.3:45745)
I20260619 16:41:11.334676 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:11.329366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40965 (local address 127.0.98.1:52347)
0619 16:41:11.329834 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:11.329850 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:11.330045 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:11.330539 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:11.330542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:11.330566 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:11.330729 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:11.330736 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.331959 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.331963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:11.333117 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:11.333126 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.334091 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.334095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:11.334219 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:11.334250 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 16:41:11.334357 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 16:41:11.334446 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":134,"threads_started":1}
I20260619 16:41:11.334772 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:11.329459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52347 (local address 127.0.98.2:40965)
0619 16:41:11.329651 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:11.329658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:11.329949 (+ 291us) server_negotiation.cc:407] Connection header received
0619 16:41:11.330223 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:11.330228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:11.330296 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:11.330390 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:11.330888 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.331842 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.333248 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.333917 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.334007 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:11.334081 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:11.334441 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 16:41:11.334593 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260619 16:41:11.335183 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:11.329723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46113 (local address 127.0.98.3:45745)
0619 16:41:11.329920 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:11.329927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:11.330195 (+ 268us) server_negotiation.cc:407] Connection header received
0619 16:41:11.330342 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:11.330345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:11.330528 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:11.330622 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:11.331559 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.332381 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.333807 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.334454 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.334567 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:11.334630 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:11.334923 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 16:41:11.335058 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 16:41:11.335427 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba77e66f2b894e1dad718bc5db89acef" candidate_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05e53be59c9f40c1806e9662993fd161" is_pre_election: true
I20260619 16:41:11.335623 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:11.329644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45745 (local address 127.0.98.1:46113)
0619 16:41:11.330031 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:11.330045 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:11.330223 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:11.330650 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:11.330652 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:11.330668 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:11.331421 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:11.331432 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.332563 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.332567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:11.333645 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:11.333655 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.334610 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.334614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:11.334729 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:11.334754 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 16:41:11.335380 (+ 626us) client_negotiation.cc:240] Negotiation successful
0619 16:41:11.335473 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":154,"threads_started":1}
I20260619 16:41:11.335626 2424 raft_consensus.cc:2454] T ba77e66f2b894e1dad718bc5db89acef P 05e53be59c9f40c1806e9662993fd161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ad5adf9d0e545eb8cccef1d682b37b9 in term 0.
I20260619 16:41:11.336189 2216 leader_election.cc:302] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05e53be59c9f40c1806e9662993fd161, 4ad5adf9d0e545eb8cccef1d682b37b9; no voters:
I20260619 16:41:11.336539 2639 raft_consensus.cc:2781] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:41:11.336624 2639 raft_consensus.cc:482] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:41:11.336529 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba77e66f2b894e1dad718bc5db89acef" candidate_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" is_pre_election: true
I20260619 16:41:11.336656 2639 raft_consensus.cc:3037] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:11.336840 2564 raft_consensus.cc:2454] T ba77e66f2b894e1dad718bc5db89acef P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ad5adf9d0e545eb8cccef1d682b37b9 in term 0.
I20260619 16:41:11.337980 2639 raft_consensus.cc:504] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:11.338428 2639 leader_election.cc:288] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9 [CANDIDATE]: Term 1 election: Requested vote from peers 05e53be59c9f40c1806e9662993fd161 (127.0.98.2:40965), 3d782f18c34e43a3b39fc58ffdc8e1d9 (127.0.98.3:45745)
I20260619 16:41:11.338940 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba77e66f2b894e1dad718bc5db89acef" candidate_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05e53be59c9f40c1806e9662993fd161"
I20260619 16:41:11.339102 2424 raft_consensus.cc:3037] T ba77e66f2b894e1dad718bc5db89acef P 05e53be59c9f40c1806e9662993fd161 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:11.339279 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba77e66f2b894e1dad718bc5db89acef" candidate_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9"
I20260619 16:41:11.339493 2564 raft_consensus.cc:3037] T ba77e66f2b894e1dad718bc5db89acef P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:11.340704 2424 raft_consensus.cc:2454] T ba77e66f2b894e1dad718bc5db89acef P 05e53be59c9f40c1806e9662993fd161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ad5adf9d0e545eb8cccef1d682b37b9 in term 1.
I20260619 16:41:11.340975 2564 raft_consensus.cc:2454] T ba77e66f2b894e1dad718bc5db89acef P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ad5adf9d0e545eb8cccef1d682b37b9 in term 1.
I20260619 16:41:11.341166 2216 leader_election.cc:302] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05e53be59c9f40c1806e9662993fd161, 4ad5adf9d0e545eb8cccef1d682b37b9; no voters:
I20260619 16:41:11.341535 2639 raft_consensus.cc:2781] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:11.342124 2639 raft_consensus.cc:686] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 1 LEADER]: Becoming Leader. State: Replica: 4ad5adf9d0e545eb8cccef1d682b37b9, State: Running, Role: LEADER
I20260619 16:41:11.342592 2639 consensus_queue.cc:237] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:11.344779 2656 tablet_replica.cc:439] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9: TxnStatusTablet state changed. Reason: New leader 4ad5adf9d0e545eb8cccef1d682b37b9. Latest consensus state: current_term: 1 leader_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } } }
I20260619 16:41:11.344946 2656 tablet_replica.cc:442] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:41:11.345405 2658 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 16:41:11.345695 2658 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 16:41:11.346760 2125 catalog_manager.cc:5495] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ad5adf9d0e545eb8cccef1d682b37b9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } health_report { overall_health: UNKNOWN } } }
I20260619 16:41:11.385206 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:11.379882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46343 (local address 127.0.0.1:55720)
0619 16:41:11.380163 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:11.380187 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:11.380406 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:11.381112 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:11.381119 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:11.381134 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:41:11.381269 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:11.381304 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.382203 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.382210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:11.382934 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:11.382946 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.383532 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.383537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:11.383679 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:11.383783 (+ 104us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:41:11.384850 (+ 1067us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:41:11.384873 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:41:11.384963 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 16:41:11.385074 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 16:41:11.385206 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:11.380038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55720 (local address 127.0.98.1:46343)
0619 16:41:11.380199 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:11.380203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:11.380307 (+ 104us) server_negotiation.cc:407] Connection header received
0619 16:41:11.380695 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:11.380698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:11.380853 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:11.381027 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:41:11.381442 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.382050 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.383111 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.383380 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.383525 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:11.383987 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:41:11.384519 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:41:11.384874 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:11.384973 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 16:41:11.385041 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 16:41:11.392153 2564 raft_consensus.cc:1261] T ba77e66f2b894e1dad718bc5db89acef P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 1 FOLLOWER]: Refusing update from remote peer 4ad5adf9d0e545eb8cccef1d682b37b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:11.392452 2424 raft_consensus.cc:1261] T ba77e66f2b894e1dad718bc5db89acef P 05e53be59c9f40c1806e9662993fd161 [term 1 FOLLOWER]: Refusing update from remote peer 4ad5adf9d0e545eb8cccef1d682b37b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:11.393002 2656 consensus_queue.cc:1035] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:11.393296 2639 consensus_queue.cc:1035] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:11.398566 2641 tablet_replica.cc:439] T ba77e66f2b894e1dad718bc5db89acef P 3d782f18c34e43a3b39fc58ffdc8e1d9: TxnStatusTablet state changed. Reason: New leader 4ad5adf9d0e545eb8cccef1d682b37b9. Latest consensus state: current_term: 1 leader_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } } }
I20260619 16:41:11.398765 2641 tablet_replica.cc:442] T ba77e66f2b894e1dad718bc5db89acef P 3d782f18c34e43a3b39fc58ffdc8e1d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:41:11.402669 2640 tablet_replica.cc:439] T ba77e66f2b894e1dad718bc5db89acef P 05e53be59c9f40c1806e9662993fd161: TxnStatusTablet state changed. Reason: New leader 4ad5adf9d0e545eb8cccef1d682b37b9. Latest consensus state: current_term: 1 leader_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } } }
I20260619 16:41:11.402835 2640 tablet_replica.cc:442] T ba77e66f2b894e1dad718bc5db89acef P 05e53be59c9f40c1806e9662993fd161: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:41:11.403906 2661 mvcc.cc:205] Tried to move back new op lower bound from 7298610263613501440 to 7298610263422283776. Current Snapshot: MvccSnapshot[applied={T|T < 7298610263613501440}]
I20260619 16:41:11.404667 2641 tablet_replica.cc:439] T ba77e66f2b894e1dad718bc5db89acef P 3d782f18c34e43a3b39fc58ffdc8e1d9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } } }
I20260619 16:41:11.404798 2641 tablet_replica.cc:442] T ba77e66f2b894e1dad718bc5db89acef P 3d782f18c34e43a3b39fc58ffdc8e1d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:41:11.405025 2656 tablet_replica.cc:439] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } } }
I20260619 16:41:11.405238 2656 tablet_replica.cc:442] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:41:11.405431 2662 mvcc.cc:205] Tried to move back new op lower bound from 7298610263613501440 to 7298610263422283776. Current Snapshot: MvccSnapshot[applied={T|T < 7298610263613501440}]
I20260619 16:41:11.406028 2639 tablet_replica.cc:439] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } } }
I20260619 16:41:11.406143 2639 tablet_replica.cc:442] T ba77e66f2b894e1dad718bc5db89acef P 4ad5adf9d0e545eb8cccef1d682b37b9: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:41:11.408864 2640 tablet_replica.cc:439] T ba77e66f2b894e1dad718bc5db89acef P 05e53be59c9f40c1806e9662993fd161: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } } }
I20260619 16:41:11.409047 2640 tablet_replica.cc:442] T ba77e66f2b894e1dad718bc5db89acef P 05e53be59c9f40c1806e9662993fd161: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:41:11.409582 2663 mvcc.cc:205] Tried to move back new op lower bound from 7298610263613501440 to 7298610263422283776. Current Snapshot: MvccSnapshot[applied={T|T < 7298610263613501440}]
I20260619 16:41:11.426281 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:11.420237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44784 (local address 127.0.98.3:45745)
0619 16:41:11.420405 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:11.420411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:11.420430 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:41:11.420624 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:11.420628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:11.420717 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:11.420876 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:41:11.421199 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.422045 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.423864 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.424219 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.424329 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:11.424803 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:41:11.425617 (+ 814us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:41:11.425761 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:11.425978 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 16:41:11.426071 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:41:11.441638 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:11.437618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57505 (local address 127.0.98.1:46343)
0619 16:41:11.437815 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:11.437820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:11.437836 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:41:11.437968 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:11.437971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:11.438027 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:11.438253 (+ 226us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:41:11.438610 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.439152 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.440280 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.440514 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.440637 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:11.441012 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:41:11.441096 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:41:11.441179 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:11.441416 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 16:41:11.441480 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260619 16:41:11.441638 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:11.437468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46343 (local address 127.0.98.3:57505)
0619 16:41:11.437719 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:11.437738 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:11.437845 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:11.438258 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:11.438261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:11.438269 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:41:11.438470 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:11.438480 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.439410 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.439414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:11.440143 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:11.440152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.440645 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.440648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:11.440743 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:11.440796 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:41:11.441186 (+ 390us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:41:11.441194 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 16:41:11.441401 (+ 207us) client_negotiation.cc:240] Negotiation successful
0619 16:41:11.441493 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 16:41:11.451431 2424 raft_consensus.cc:1261] T d4fbebc2ae1b494f8f57116e63e6a55a P 05e53be59c9f40c1806e9662993fd161 [term 1 FOLLOWER]: Refusing update from remote peer 3d782f18c34e43a3b39fc58ffdc8e1d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:11.451520 2284 raft_consensus.cc:1261] T d4fbebc2ae1b494f8f57116e63e6a55a P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 1 FOLLOWER]: Refusing update from remote peer 3d782f18c34e43a3b39fc58ffdc8e1d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:11.452183 2643 consensus_queue.cc:1035] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:11.452421 2641 consensus_queue.cc:1035] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:11.485132 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:11.480081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55203 (local address 127.0.98.3:45745)
0619 16:41:11.480257 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:11.480263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:11.480645 (+ 382us) server_negotiation.cc:407] Connection header received
0619 16:41:11.480832 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:11.480835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:11.480904 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:11.480996 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:41:11.481408 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.482307 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.483356 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.483788 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.483925 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:11.484490 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:41:11.484612 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:41:11.484698 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:11.484865 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 16:41:11.484943 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 16:41:11.485682 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:11.479926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45745 (local address 127.0.98.1:55203)
0619 16:41:11.480531 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:11.480548 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:11.480673 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:11.481041 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:11.481044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:11.481052 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:41:11.481202 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:11.481211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.482472 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.482475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:11.483216 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:11.483225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:11.484051 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:11.484056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:11.484232 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:11.484310 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:41:11.484736 (+ 426us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:41:11.484744 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 16:41:11.484828 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 16:41:11.485063 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":136,"threads_started":1}
I20260619 16:41:11.520358 2125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:41:11.520668 2125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:41:11.524583 2125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e74dc107d0148a4a928ae1bce521b71: Sending DeleteTablet for 3 replicas of tablet d4fbebc2ae1b494f8f57116e63e6a55a
I20260619 16:41:11.525525 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet d4fbebc2ae1b494f8f57116e63e6a55a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55712
I20260619 16:41:11.526237 2679 tablet_replica.cc:330] T d4fbebc2ae1b494f8f57116e63e6a55a P 4ad5adf9d0e545eb8cccef1d682b37b9: stopping tablet replica
I20260619 16:41:11.526631 2544 tablet_service.cc:1514] Processing DeleteTablet for tablet d4fbebc2ae1b494f8f57116e63e6a55a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44770
I20260619 16:41:11.526695 2679 raft_consensus.cc:2229] T d4fbebc2ae1b494f8f57116e63e6a55a P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:41:11.526912 2679 raft_consensus.cc:2258] T d4fbebc2ae1b494f8f57116e63e6a55a P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:11.528174 2404 tablet_service.cc:1514] Processing DeleteTablet for tablet d4fbebc2ae1b494f8f57116e63e6a55a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34622
I20260619 16:41:11.528720 2679 ts_tablet_manager.cc:1905] T d4fbebc2ae1b494f8f57116e63e6a55a P 4ad5adf9d0e545eb8cccef1d682b37b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:11.529827 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:41:11.530332 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:41:11.532791 2679 ts_tablet_manager.cc:1918] T d4fbebc2ae1b494f8f57116e63e6a55a P 4ad5adf9d0e545eb8cccef1d682b37b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:41:11.532976 2679 log.cc:1196] T d4fbebc2ae1b494f8f57116e63e6a55a P 4ad5adf9d0e545eb8cccef1d682b37b9: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-0/wal/wals/d4fbebc2ae1b494f8f57116e63e6a55a
I20260619 16:41:11.533399 2679 ts_tablet_manager.cc:1939] T d4fbebc2ae1b494f8f57116e63e6a55a P 4ad5adf9d0e545eb8cccef1d682b37b9: Deleting consensus metadata
I20260619 16:41:11.535254 2113 catalog_manager.cc:4841] TS 4ad5adf9d0e545eb8cccef1d682b37b9 (127.0.98.1:46343): tablet d4fbebc2ae1b494f8f57116e63e6a55a (table test-table [id=503fc2ac80fc4822bb0eb44e31c0543f]) successfully deleted
I20260619 16:41:11.536474 2404 tablet_service.cc:1467] Processing CreateTablet for tablet 095cac1da53e47b789d19ffec3769714 (DEFAULT_TABLE table=test-table [id=9d60effbc28941f38e645b97b745639c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:11.536855 2404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 095cac1da53e47b789d19ffec3769714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:11.536829 2544 tablet_service.cc:1467] Processing CreateTablet for tablet 095cac1da53e47b789d19ffec3769714 (DEFAULT_TABLE table=test-table [id=9d60effbc28941f38e645b97b745639c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:11.537171 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 095cac1da53e47b789d19ffec3769714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:11.539959 2634 tablet_bootstrap.cc:492] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9: Bootstrap starting.
I20260619 16:41:11.540509 2264 tablet_service.cc:1467] Processing CreateTablet for tablet 095cac1da53e47b789d19ffec3769714 (DEFAULT_TABLE table=test-table [id=9d60effbc28941f38e645b97b745639c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:11.540875 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 095cac1da53e47b789d19ffec3769714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:11.541040 2634 tablet_bootstrap.cc:654] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:11.543406 2681 tablet_replica.cc:330] T d4fbebc2ae1b494f8f57116e63e6a55a P 05e53be59c9f40c1806e9662993fd161: stopping tablet replica
I20260619 16:41:11.543891 2633 tablet_bootstrap.cc:492] T 095cac1da53e47b789d19ffec3769714 P 4ad5adf9d0e545eb8cccef1d682b37b9: Bootstrap starting.
I20260619 16:41:11.545047 2633 tablet_bootstrap.cc:654] T 095cac1da53e47b789d19ffec3769714 P 4ad5adf9d0e545eb8cccef1d682b37b9: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:11.545382 2681 raft_consensus.cc:2229] T d4fbebc2ae1b494f8f57116e63e6a55a P 05e53be59c9f40c1806e9662993fd161 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:41:11.545702 2680 tablet_replica.cc:330] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9: stopping tablet replica
I20260619 16:41:11.546518 2635 tablet_bootstrap.cc:492] T 095cac1da53e47b789d19ffec3769714 P 05e53be59c9f40c1806e9662993fd161: Bootstrap starting.
I20260619 16:41:11.546777 2681 raft_consensus.cc:2258] T d4fbebc2ae1b494f8f57116e63e6a55a P 05e53be59c9f40c1806e9662993fd161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:11.548157 2680 raft_consensus.cc:2229] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:41:11.547951 2635 tablet_bootstrap.cc:654] T 095cac1da53e47b789d19ffec3769714 P 05e53be59c9f40c1806e9662993fd161: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:11.549546 2634 tablet_bootstrap.cc:492] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9: No bootstrap required, opened a new log
I20260619 16:41:11.549682 2634 ts_tablet_manager.cc:1397] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260619 16:41:11.550316 2634 raft_consensus.cc:348] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:11.550468 2634 raft_consensus.cc:374] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:11.550498 2634 raft_consensus.cc:729] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d782f18c34e43a3b39fc58ffdc8e1d9, State: Initialized, Role: FOLLOWER
I20260619 16:41:11.550632 2634 consensus_queue.cc:260] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:11.550910 2634 ts_tablet_manager.cc:1428] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:41:11.552763 2681 ts_tablet_manager.cc:1905] T d4fbebc2ae1b494f8f57116e63e6a55a P 05e53be59c9f40c1806e9662993fd161: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:11.549863 2680 raft_consensus.cc:2258] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:11.555881 2680 ts_tablet_manager.cc:1905] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:11.557718 2633 tablet_bootstrap.cc:492] T 095cac1da53e47b789d19ffec3769714 P 4ad5adf9d0e545eb8cccef1d682b37b9: No bootstrap required, opened a new log
I20260619 16:41:11.557869 2633 ts_tablet_manager.cc:1397] T 095cac1da53e47b789d19ffec3769714 P 4ad5adf9d0e545eb8cccef1d682b37b9: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20260619 16:41:11.558472 2633 raft_consensus.cc:348] T 095cac1da53e47b789d19ffec3769714 P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:11.558640 2633 raft_consensus.cc:374] T 095cac1da53e47b789d19ffec3769714 P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:11.558676 2633 raft_consensus.cc:729] T 095cac1da53e47b789d19ffec3769714 P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ad5adf9d0e545eb8cccef1d682b37b9, State: Initialized, Role: FOLLOWER
I20260619 16:41:11.558820 2633 consensus_queue.cc:260] T 095cac1da53e47b789d19ffec3769714 P 4ad5adf9d0e545eb8cccef1d682b37b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:11.559167 2633 ts_tablet_manager.cc:1428] T 095cac1da53e47b789d19ffec3769714 P 4ad5adf9d0e545eb8cccef1d682b37b9: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260619 16:41:11.561028 2681 ts_tablet_manager.cc:1918] T d4fbebc2ae1b494f8f57116e63e6a55a P 05e53be59c9f40c1806e9662993fd161: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:41:11.561161 2681 log.cc:1196] T d4fbebc2ae1b494f8f57116e63e6a55a P 05e53be59c9f40c1806e9662993fd161: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-1/wal/wals/d4fbebc2ae1b494f8f57116e63e6a55a
I20260619 16:41:11.561548 2681 ts_tablet_manager.cc:1939] T d4fbebc2ae1b494f8f57116e63e6a55a P 05e53be59c9f40c1806e9662993fd161: Deleting consensus metadata
I20260619 16:41:11.562067 2680 ts_tablet_manager.cc:1918] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:41:11.562281 2680 log.cc:1196] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-2/wal/wals/d4fbebc2ae1b494f8f57116e63e6a55a
I20260619 16:41:11.562817 2680 ts_tablet_manager.cc:1939] T d4fbebc2ae1b494f8f57116e63e6a55a P 3d782f18c34e43a3b39fc58ffdc8e1d9: Deleting consensus metadata
I20260619 16:41:11.563295 2112 catalog_manager.cc:4841] TS 05e53be59c9f40c1806e9662993fd161 (127.0.98.2:40965): tablet d4fbebc2ae1b494f8f57116e63e6a55a (table test-table [id=503fc2ac80fc4822bb0eb44e31c0543f]) successfully deleted
I20260619 16:41:11.564564 2635 tablet_bootstrap.cc:492] T 095cac1da53e47b789d19ffec3769714 P 05e53be59c9f40c1806e9662993fd161: No bootstrap required, opened a new log
I20260619 16:41:11.564708 2635 ts_tablet_manager.cc:1397] T 095cac1da53e47b789d19ffec3769714 P 05e53be59c9f40c1806e9662993fd161: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20260619 16:41:11.565265 2635 raft_consensus.cc:348] T 095cac1da53e47b789d19ffec3769714 P 05e53be59c9f40c1806e9662993fd161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:11.565407 2635 raft_consensus.cc:374] T 095cac1da53e47b789d19ffec3769714 P 05e53be59c9f40c1806e9662993fd161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:11.565442 2635 raft_consensus.cc:729] T 095cac1da53e47b789d19ffec3769714 P 05e53be59c9f40c1806e9662993fd161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05e53be59c9f40c1806e9662993fd161, State: Initialized, Role: FOLLOWER
I20260619 16:41:11.565582 2635 consensus_queue.cc:260] T 095cac1da53e47b789d19ffec3769714 P 05e53be59c9f40c1806e9662993fd161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:11.565888 2635 ts_tablet_manager.cc:1428] T 095cac1da53e47b789d19ffec3769714 P 05e53be59c9f40c1806e9662993fd161: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:41:11.567098 2113 catalog_manager.cc:4841] TS 3d782f18c34e43a3b39fc58ffdc8e1d9 (127.0.98.3:45745): tablet d4fbebc2ae1b494f8f57116e63e6a55a (table test-table [id=503fc2ac80fc4822bb0eb44e31c0543f]) successfully deleted
I20260619 16:41:11.610546 2643 raft_consensus.cc:482] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:41:11.610767 2643 raft_consensus.cc:504] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:11.611395 2643 leader_election.cc:288] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ad5adf9d0e545eb8cccef1d682b37b9 (127.0.98.1:46343), 05e53be59c9f40c1806e9662993fd161 (127.0.98.2:40965)
I20260619 16:41:11.612037 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "095cac1da53e47b789d19ffec3769714" candidate_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05e53be59c9f40c1806e9662993fd161" is_pre_election: true
I20260619 16:41:11.612056 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "095cac1da53e47b789d19ffec3769714" candidate_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" is_pre_election: true
I20260619 16:41:11.612226 2284 raft_consensus.cc:2454] T 095cac1da53e47b789d19ffec3769714 P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d782f18c34e43a3b39fc58ffdc8e1d9 in term 0.
I20260619 16:41:11.612227 2423 raft_consensus.cc:2454] T 095cac1da53e47b789d19ffec3769714 P 05e53be59c9f40c1806e9662993fd161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d782f18c34e43a3b39fc58ffdc8e1d9 in term 0.
I20260619 16:41:11.612596 2496 leader_election.cc:302] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05e53be59c9f40c1806e9662993fd161, 3d782f18c34e43a3b39fc58ffdc8e1d9; no voters:
I20260619 16:41:11.612790 2643 raft_consensus.cc:2781] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:41:11.612855 2643 raft_consensus.cc:482] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:41:11.612901 2643 raft_consensus.cc:3037] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:11.614310 2643 raft_consensus.cc:504] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:11.614763 2643 leader_election.cc:288] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4ad5adf9d0e545eb8cccef1d682b37b9 (127.0.98.1:46343), 05e53be59c9f40c1806e9662993fd161 (127.0.98.2:40965)
I20260619 16:41:11.615227 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "095cac1da53e47b789d19ffec3769714" candidate_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9"
I20260619 16:41:11.615429 2284 raft_consensus.cc:3037] T 095cac1da53e47b789d19ffec3769714 P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:11.615368 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "095cac1da53e47b789d19ffec3769714" candidate_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05e53be59c9f40c1806e9662993fd161"
I20260619 16:41:11.615563 2423 raft_consensus.cc:3037] T 095cac1da53e47b789d19ffec3769714 P 05e53be59c9f40c1806e9662993fd161 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:11.616741 2284 raft_consensus.cc:2454] T 095cac1da53e47b789d19ffec3769714 P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d782f18c34e43a3b39fc58ffdc8e1d9 in term 1.
I20260619 16:41:11.617067 2423 raft_consensus.cc:2454] T 095cac1da53e47b789d19ffec3769714 P 05e53be59c9f40c1806e9662993fd161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d782f18c34e43a3b39fc58ffdc8e1d9 in term 1.
I20260619 16:41:11.617166 2497 leader_election.cc:302] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d782f18c34e43a3b39fc58ffdc8e1d9, 4ad5adf9d0e545eb8cccef1d682b37b9; no voters:
I20260619 16:41:11.617962 2643 raft_consensus.cc:2781] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:11.618108 2643 raft_consensus.cc:686] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 1 LEADER]: Becoming Leader. State: Replica: 3d782f18c34e43a3b39fc58ffdc8e1d9, State: Running, Role: LEADER
I20260619 16:41:11.618578 2643 consensus_queue.cc:237] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:11.621167 2125 catalog_manager.cc:5495] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d782f18c34e43a3b39fc58ffdc8e1d9 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } health_report { overall_health: HEALTHY } } }
I20260619 16:41:11.651235 2423 raft_consensus.cc:1261] T 095cac1da53e47b789d19ffec3769714 P 05e53be59c9f40c1806e9662993fd161 [term 1 FOLLOWER]: Refusing update from remote peer 3d782f18c34e43a3b39fc58ffdc8e1d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:11.651453 2284 raft_consensus.cc:1261] T 095cac1da53e47b789d19ffec3769714 P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 1 FOLLOWER]: Refusing update from remote peer 3d782f18c34e43a3b39fc58ffdc8e1d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:11.652060 2641 consensus_queue.cc:1035] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:11.652379 2643 consensus_queue.cc:1035] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:41:11.675761 2662 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 16:41:13.268693 2126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:41:13.268919 2126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:41:13.272850 2126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e74dc107d0148a4a928ae1bce521b71: Sending DeleteTablet for 3 replicas of tablet 095cac1da53e47b789d19ffec3769714
I20260619 16:41:13.273756 2404 tablet_service.cc:1514] Processing DeleteTablet for tablet 095cac1da53e47b789d19ffec3769714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34622
I20260619 16:41:13.273746 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet 095cac1da53e47b789d19ffec3769714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55712
I20260619 16:41:13.273846 2544 tablet_service.cc:1514] Processing DeleteTablet for tablet 095cac1da53e47b789d19ffec3769714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44770
I20260619 16:41:13.274220 2713 tablet_replica.cc:330] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9: stopping tablet replica
I20260619 16:41:13.274271 2714 tablet_replica.cc:330] T 095cac1da53e47b789d19ffec3769714 P 4ad5adf9d0e545eb8cccef1d682b37b9: stopping tablet replica
I20260619 16:41:13.274425 2713 raft_consensus.cc:2229] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:41:13.274430 2714 raft_consensus.cc:2229] T 095cac1da53e47b789d19ffec3769714 P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:41:13.274708 2714 raft_consensus.cc:2258] T 095cac1da53e47b789d19ffec3769714 P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:13.274763 2713 raft_consensus.cc:2258] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:13.275544 2713 ts_tablet_manager.cc:1905] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:13.275581 2714 ts_tablet_manager.cc:1905] T 095cac1da53e47b789d19ffec3769714 P 4ad5adf9d0e545eb8cccef1d682b37b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:13.278450 2713 ts_tablet_manager.cc:1918] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:41:13.278569 2713 log.cc:1196] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-2/wal/wals/095cac1da53e47b789d19ffec3769714
I20260619 16:41:13.278914 2713 ts_tablet_manager.cc:1939] T 095cac1da53e47b789d19ffec3769714 P 3d782f18c34e43a3b39fc58ffdc8e1d9: Deleting consensus metadata
I20260619 16:41:13.278935 2712 tablet_replica.cc:330] T 095cac1da53e47b789d19ffec3769714 P 05e53be59c9f40c1806e9662993fd161: stopping tablet replica
I20260619 16:41:13.279273 2712 raft_consensus.cc:2229] T 095cac1da53e47b789d19ffec3769714 P 05e53be59c9f40c1806e9662993fd161 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:41:13.279518 2712 raft_consensus.cc:2258] T 095cac1da53e47b789d19ffec3769714 P 05e53be59c9f40c1806e9662993fd161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:13.279909 2113 catalog_manager.cc:4841] TS 3d782f18c34e43a3b39fc58ffdc8e1d9 (127.0.98.3:45745): tablet 095cac1da53e47b789d19ffec3769714 (table test-table [id=9d60effbc28941f38e645b97b745639c]) successfully deleted
I20260619 16:41:13.280751 2712 ts_tablet_manager.cc:1905] T 095cac1da53e47b789d19ffec3769714 P 05e53be59c9f40c1806e9662993fd161: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:13.281204 2714 ts_tablet_manager.cc:1918] T 095cac1da53e47b789d19ffec3769714 P 4ad5adf9d0e545eb8cccef1d682b37b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:41:13.281319 2714 log.cc:1196] T 095cac1da53e47b789d19ffec3769714 P 4ad5adf9d0e545eb8cccef1d682b37b9: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-0/wal/wals/095cac1da53e47b789d19ffec3769714
I20260619 16:41:13.281653 2714 ts_tablet_manager.cc:1939] T 095cac1da53e47b789d19ffec3769714 P 4ad5adf9d0e545eb8cccef1d682b37b9: Deleting consensus metadata
I20260619 16:41:13.282650 2113 catalog_manager.cc:4841] TS 4ad5adf9d0e545eb8cccef1d682b37b9 (127.0.98.1:46343): tablet 095cac1da53e47b789d19ffec3769714 (table test-table [id=9d60effbc28941f38e645b97b745639c]) successfully deleted
I20260619 16:41:13.284189 2712 ts_tablet_manager.cc:1918] T 095cac1da53e47b789d19ffec3769714 P 05e53be59c9f40c1806e9662993fd161: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:41:13.284294 2712 log.cc:1196] T 095cac1da53e47b789d19ffec3769714 P 05e53be59c9f40c1806e9662993fd161: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-1/wal/wals/095cac1da53e47b789d19ffec3769714
I20260619 16:41:13.284592 2712 ts_tablet_manager.cc:1939] T 095cac1da53e47b789d19ffec3769714 P 05e53be59c9f40c1806e9662993fd161: Deleting consensus metadata
I20260619 16:41:13.285899 2112 catalog_manager.cc:4841] TS 05e53be59c9f40c1806e9662993fd161 (127.0.98.2:40965): tablet 095cac1da53e47b789d19ffec3769714 (table test-table [id=9d60effbc28941f38e645b97b745639c]) successfully deleted
Jun 19 16:41:13 dist-test-slave-03cd krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887273, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:41:13 dist-test-slave-03cd krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887273, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:13.312254 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:13.298774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50870 (local address 127.0.98.62:38541)
0619 16:41:13.299122 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:13.299130 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:13.299212 (+ 82us) server_negotiation.cc:407] Connection header received
0619 16:41:13.299475 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:13.299479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:13.299544 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:13.299752 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:13.300579 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:13.301191 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:13.302768 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:13.303043 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:13.303218 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:13.307316 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:13.307322 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:13.307325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:13.307377 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:13.309711 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:13.310432 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:13.310435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:13.310437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:13.310554 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:13.310888 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:13.310890 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:13.310892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:13.311134 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:13.311308 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:13.311645 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 16:41:13.311939 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":165,"threads_started":1}
I20260619 16:41:13.316886 2126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:41:13.317836 2126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:41:13.327209 2544 tablet_service.cc:1467] Processing CreateTablet for tablet 0d82fa7d9fe34f0ca330900583fdb285 (DEFAULT_TABLE table=test-table [id=40cabd47627f4461bf295048296d3ce6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:13.327209 2264 tablet_service.cc:1467] Processing CreateTablet for tablet 0d82fa7d9fe34f0ca330900583fdb285 (DEFAULT_TABLE table=test-table [id=40cabd47627f4461bf295048296d3ce6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:13.327214 2404 tablet_service.cc:1467] Processing CreateTablet for tablet 0d82fa7d9fe34f0ca330900583fdb285 (DEFAULT_TABLE table=test-table [id=40cabd47627f4461bf295048296d3ce6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:13.327633 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d82fa7d9fe34f0ca330900583fdb285. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:13.327682 2404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d82fa7d9fe34f0ca330900583fdb285. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:13.327633 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d82fa7d9fe34f0ca330900583fdb285. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:13.330677 2723 tablet_bootstrap.cc:492] T 0d82fa7d9fe34f0ca330900583fdb285 P 05e53be59c9f40c1806e9662993fd161: Bootstrap starting.
I20260619 16:41:13.330861 2724 tablet_bootstrap.cc:492] T 0d82fa7d9fe34f0ca330900583fdb285 P 3d782f18c34e43a3b39fc58ffdc8e1d9: Bootstrap starting.
I20260619 16:41:13.331280 2725 tablet_bootstrap.cc:492] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9: Bootstrap starting.
I20260619 16:41:13.332059 2724 tablet_bootstrap.cc:654] T 0d82fa7d9fe34f0ca330900583fdb285 P 3d782f18c34e43a3b39fc58ffdc8e1d9: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:13.332059 2723 tablet_bootstrap.cc:654] T 0d82fa7d9fe34f0ca330900583fdb285 P 05e53be59c9f40c1806e9662993fd161: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:13.332504 2725 tablet_bootstrap.cc:654] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:13.334128 2724 tablet_bootstrap.cc:492] T 0d82fa7d9fe34f0ca330900583fdb285 P 3d782f18c34e43a3b39fc58ffdc8e1d9: No bootstrap required, opened a new log
I20260619 16:41:13.334270 2724 ts_tablet_manager.cc:1397] T 0d82fa7d9fe34f0ca330900583fdb285 P 3d782f18c34e43a3b39fc58ffdc8e1d9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:41:13.334343 2725 tablet_bootstrap.cc:492] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9: No bootstrap required, opened a new log
I20260619 16:41:13.334553 2725 ts_tablet_manager.cc:1397] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:41:13.334568 2723 tablet_bootstrap.cc:492] T 0d82fa7d9fe34f0ca330900583fdb285 P 05e53be59c9f40c1806e9662993fd161: No bootstrap required, opened a new log
I20260619 16:41:13.334659 2723 ts_tablet_manager.cc:1397] T 0d82fa7d9fe34f0ca330900583fdb285 P 05e53be59c9f40c1806e9662993fd161: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 16:41:13.334890 2724 raft_consensus.cc:348] T 0d82fa7d9fe34f0ca330900583fdb285 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:13.335043 2724 raft_consensus.cc:374] T 0d82fa7d9fe34f0ca330900583fdb285 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:13.335084 2724 raft_consensus.cc:729] T 0d82fa7d9fe34f0ca330900583fdb285 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d782f18c34e43a3b39fc58ffdc8e1d9, State: Initialized, Role: FOLLOWER
I20260619 16:41:13.335110 2723 raft_consensus.cc:348] T 0d82fa7d9fe34f0ca330900583fdb285 P 05e53be59c9f40c1806e9662993fd161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:13.335110 2725 raft_consensus.cc:348] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:13.335247 2725 raft_consensus.cc:374] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:13.335247 2723 raft_consensus.cc:374] T 0d82fa7d9fe34f0ca330900583fdb285 P 05e53be59c9f40c1806e9662993fd161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:13.335295 2723 raft_consensus.cc:729] T 0d82fa7d9fe34f0ca330900583fdb285 P 05e53be59c9f40c1806e9662993fd161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05e53be59c9f40c1806e9662993fd161, State: Initialized, Role: FOLLOWER
I20260619 16:41:13.335295 2725 raft_consensus.cc:729] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ad5adf9d0e545eb8cccef1d682b37b9, State: Initialized, Role: FOLLOWER
I20260619 16:41:13.335250 2724 consensus_queue.cc:260] T 0d82fa7d9fe34f0ca330900583fdb285 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:13.335590 2724 ts_tablet_manager.cc:1428] T 0d82fa7d9fe34f0ca330900583fdb285 P 3d782f18c34e43a3b39fc58ffdc8e1d9: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 16:41:13.335458 2725 consensus_queue.cc:260] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:13.335500 2723 consensus_queue.cc:260] T 0d82fa7d9fe34f0ca330900583fdb285 P 05e53be59c9f40c1806e9662993fd161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:13.336184 2725 ts_tablet_manager.cc:1428] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:41:13.336370 2723 ts_tablet_manager.cc:1428] T 0d82fa7d9fe34f0ca330900583fdb285 P 05e53be59c9f40c1806e9662993fd161: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 16:41:13.543608 2693 raft_consensus.cc:482] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:41:13.543912 2693 raft_consensus.cc:504] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:13.544541 2693 leader_election.cc:288] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05e53be59c9f40c1806e9662993fd161 (127.0.98.2:40965), 3d782f18c34e43a3b39fc58ffdc8e1d9 (127.0.98.3:45745)
I20260619 16:41:13.545193 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d82fa7d9fe34f0ca330900583fdb285" candidate_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" is_pre_election: true
I20260619 16:41:13.545239 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d82fa7d9fe34f0ca330900583fdb285" candidate_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05e53be59c9f40c1806e9662993fd161" is_pre_election: true
I20260619 16:41:13.545420 2564 raft_consensus.cc:2454] T 0d82fa7d9fe34f0ca330900583fdb285 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ad5adf9d0e545eb8cccef1d682b37b9 in term 0.
I20260619 16:41:13.545431 2424 raft_consensus.cc:2454] T 0d82fa7d9fe34f0ca330900583fdb285 P 05e53be59c9f40c1806e9662993fd161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ad5adf9d0e545eb8cccef1d682b37b9 in term 0.
I20260619 16:41:13.545830 2217 leader_election.cc:302] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d782f18c34e43a3b39fc58ffdc8e1d9, 4ad5adf9d0e545eb8cccef1d682b37b9; no voters:
I20260619 16:41:13.546067 2693 raft_consensus.cc:2781] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:41:13.546159 2693 raft_consensus.cc:482] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:41:13.546213 2693 raft_consensus.cc:3037] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:13.547426 2693 raft_consensus.cc:504] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:13.547945 2693 leader_election.cc:288] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9 [CANDIDATE]: Term 1 election: Requested vote from peers 05e53be59c9f40c1806e9662993fd161 (127.0.98.2:40965), 3d782f18c34e43a3b39fc58ffdc8e1d9 (127.0.98.3:45745)
I20260619 16:41:13.548377 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d82fa7d9fe34f0ca330900583fdb285" candidate_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05e53be59c9f40c1806e9662993fd161"
I20260619 16:41:13.548434 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d82fa7d9fe34f0ca330900583fdb285" candidate_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9"
I20260619 16:41:13.548544 2564 raft_consensus.cc:3037] T 0d82fa7d9fe34f0ca330900583fdb285 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:13.548543 2424 raft_consensus.cc:3037] T 0d82fa7d9fe34f0ca330900583fdb285 P 05e53be59c9f40c1806e9662993fd161 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:13.549772 2564 raft_consensus.cc:2454] T 0d82fa7d9fe34f0ca330900583fdb285 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ad5adf9d0e545eb8cccef1d682b37b9 in term 1.
I20260619 16:41:13.549772 2424 raft_consensus.cc:2454] T 0d82fa7d9fe34f0ca330900583fdb285 P 05e53be59c9f40c1806e9662993fd161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ad5adf9d0e545eb8cccef1d682b37b9 in term 1.
I20260619 16:41:13.550160 2216 leader_election.cc:302] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05e53be59c9f40c1806e9662993fd161, 4ad5adf9d0e545eb8cccef1d682b37b9; no voters:
I20260619 16:41:13.550426 2693 raft_consensus.cc:2781] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:13.550550 2693 raft_consensus.cc:686] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 1 LEADER]: Becoming Leader. State: Replica: 4ad5adf9d0e545eb8cccef1d682b37b9, State: Running, Role: LEADER
I20260619 16:41:13.550736 2693 consensus_queue.cc:237] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:13.552980 2125 catalog_manager.cc:5495] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ad5adf9d0e545eb8cccef1d682b37b9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } health_report { overall_health: UNKNOWN } } }
I20260619 16:41:13.589331 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:13.585765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55722 (local address 127.0.98.1:46343)
0619 16:41:13.586184 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:13.586191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:13.586202 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:41:13.586241 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:13.586243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:13.586285 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:13.586382 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:41:13.586704 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:13.587322 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:13.588201 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:13.588445 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:13.588560 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:13.588823 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:41:13.588906 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:41:13.588999 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:13.589108 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 16:41:13.589163 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":191,"threads_started":1}
I20260619 16:41:13.601313 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:13.597515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59189 (local address 127.0.98.1:46343)
0619 16:41:13.597755 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:13.597760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:13.597953 (+ 193us) server_negotiation.cc:407] Connection header received
0619 16:41:13.598093 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:13.598095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:13.598153 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:13.598229 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:41:13.598597 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:13.599166 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:13.600183 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:13.600405 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:13.600488 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:13.600749 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:41:13.600837 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:41:13.600898 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:13.601087 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 16:41:13.601143 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260619 16:41:13.601332 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:13.597411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46343 (local address 127.0.98.1:59189)
0619 16:41:13.597792 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:13.597808 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:13.597989 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:13.598304 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:13.598307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:13.598317 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:41:13.598432 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:13.598436 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:13.599293 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:13.599296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:13.600062 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:13.600073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:13.600539 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:13.600542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:13.600607 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:13.600639 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:41:13.600976 (+ 337us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:41:13.600984 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 16:41:13.601066 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 16:41:13.601148 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":145,"threads_started":1}
I20260619 16:41:13.608711 2564 raft_consensus.cc:1261] T 0d82fa7d9fe34f0ca330900583fdb285 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 1 FOLLOWER]: Refusing update from remote peer 4ad5adf9d0e545eb8cccef1d682b37b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:13.608807 2424 raft_consensus.cc:1261] T 0d82fa7d9fe34f0ca330900583fdb285 P 05e53be59c9f40c1806e9662993fd161 [term 1 FOLLOWER]: Refusing update from remote peer 4ad5adf9d0e545eb8cccef1d682b37b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:13.609290 2706 consensus_queue.cc:1035] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:13.609519 2693 consensus_queue.cc:1035] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:13.653932 2127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:41:13.654201 2127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:41:13.657001 2127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e74dc107d0148a4a928ae1bce521b71: Sending DeleteTablet for 3 replicas of tablet 0d82fa7d9fe34f0ca330900583fdb285
I20260619 16:41:13.657846 2404 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d82fa7d9fe34f0ca330900583fdb285 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34622
I20260619 16:41:13.657846 2544 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d82fa7d9fe34f0ca330900583fdb285 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44770
I20260619 16:41:13.658089 2713 tablet_replica.cc:330] T 0d82fa7d9fe34f0ca330900583fdb285 P 3d782f18c34e43a3b39fc58ffdc8e1d9: stopping tablet replica
I20260619 16:41:13.658089 2712 tablet_replica.cc:330] T 0d82fa7d9fe34f0ca330900583fdb285 P 05e53be59c9f40c1806e9662993fd161: stopping tablet replica
I20260619 16:41:13.658215 2712 raft_consensus.cc:2229] T 0d82fa7d9fe34f0ca330900583fdb285 P 05e53be59c9f40c1806e9662993fd161 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:41:13.658215 2713 raft_consensus.cc:2229] T 0d82fa7d9fe34f0ca330900583fdb285 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:41:13.658339 2712 raft_consensus.cc:2258] T 0d82fa7d9fe34f0ca330900583fdb285 P 05e53be59c9f40c1806e9662993fd161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:13.658344 2713 raft_consensus.cc:2258] T 0d82fa7d9fe34f0ca330900583fdb285 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:13.658979 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d82fa7d9fe34f0ca330900583fdb285 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55712
I20260619 16:41:13.659112 2712 ts_tablet_manager.cc:1905] T 0d82fa7d9fe34f0ca330900583fdb285 P 05e53be59c9f40c1806e9662993fd161: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:13.659178 2714 tablet_replica.cc:330] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9: stopping tablet replica
I20260619 16:41:13.659304 2714 raft_consensus.cc:2229] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:41:13.659333 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 16:41:13.659686 2714 raft_consensus.cc:2258] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 16:41:13.660231 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:41:13.660455 2714 ts_tablet_manager.cc:1905] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:13.662072 2712 ts_tablet_manager.cc:1918] T 0d82fa7d9fe34f0ca330900583fdb285 P 05e53be59c9f40c1806e9662993fd161: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:41:13.662737 2712 log.cc:1196] T 0d82fa7d9fe34f0ca330900583fdb285 P 05e53be59c9f40c1806e9662993fd161: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-1/wal/wals/0d82fa7d9fe34f0ca330900583fdb285
I20260619 16:41:13.663041 2712 ts_tablet_manager.cc:1939] T 0d82fa7d9fe34f0ca330900583fdb285 P 05e53be59c9f40c1806e9662993fd161: Deleting consensus metadata
I20260619 16:41:13.663332 2714 ts_tablet_manager.cc:1918] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:41:13.663416 2714 log.cc:1196] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-0/wal/wals/0d82fa7d9fe34f0ca330900583fdb285
I20260619 16:41:13.663729 2714 ts_tablet_manager.cc:1939] T 0d82fa7d9fe34f0ca330900583fdb285 P 4ad5adf9d0e545eb8cccef1d682b37b9: Deleting consensus metadata
I20260619 16:41:13.664363 2713 ts_tablet_manager.cc:1905] T 0d82fa7d9fe34f0ca330900583fdb285 P 3d782f18c34e43a3b39fc58ffdc8e1d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:13.666522 2112 catalog_manager.cc:4841] TS 05e53be59c9f40c1806e9662993fd161 (127.0.98.2:40965): tablet 0d82fa7d9fe34f0ca330900583fdb285 (table test-table [id=40cabd47627f4461bf295048296d3ce6]) successfully deleted
I20260619 16:41:13.667805 2544 tablet_service.cc:1467] Processing CreateTablet for tablet 8e5e8f385dbe405aa033952fd9c5b735 (DEFAULT_TABLE table=test-table [id=e82c2f33c28d41bdb2fd1642b2ab88d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:13.667805 2404 tablet_service.cc:1467] Processing CreateTablet for tablet 8e5e8f385dbe405aa033952fd9c5b735 (DEFAULT_TABLE table=test-table [id=e82c2f33c28d41bdb2fd1642b2ab88d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:13.668111 2404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e5e8f385dbe405aa033952fd9c5b735. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:13.668133 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e5e8f385dbe405aa033952fd9c5b735. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:13.670469 2723 tablet_bootstrap.cc:492] T 8e5e8f385dbe405aa033952fd9c5b735 P 05e53be59c9f40c1806e9662993fd161: Bootstrap starting.
I20260619 16:41:13.670539 2724 tablet_bootstrap.cc:492] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9: Bootstrap starting.
I20260619 16:41:13.671598 2723 tablet_bootstrap.cc:654] T 8e5e8f385dbe405aa033952fd9c5b735 P 05e53be59c9f40c1806e9662993fd161: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:13.671599 2724 tablet_bootstrap.cc:654] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:13.672839 2264 tablet_service.cc:1467] Processing CreateTablet for tablet 8e5e8f385dbe405aa033952fd9c5b735 (DEFAULT_TABLE table=test-table [id=e82c2f33c28d41bdb2fd1642b2ab88d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:13.673188 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e5e8f385dbe405aa033952fd9c5b735. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:13.673372 2713 ts_tablet_manager.cc:1918] T 0d82fa7d9fe34f0ca330900583fdb285 P 3d782f18c34e43a3b39fc58ffdc8e1d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:41:13.673497 2713 log.cc:1196] T 0d82fa7d9fe34f0ca330900583fdb285 P 3d782f18c34e43a3b39fc58ffdc8e1d9: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-2/wal/wals/0d82fa7d9fe34f0ca330900583fdb285
I20260619 16:41:13.673770 2713 ts_tablet_manager.cc:1939] T 0d82fa7d9fe34f0ca330900583fdb285 P 3d782f18c34e43a3b39fc58ffdc8e1d9: Deleting consensus metadata
I20260619 16:41:13.675213 2723 tablet_bootstrap.cc:492] T 8e5e8f385dbe405aa033952fd9c5b735 P 05e53be59c9f40c1806e9662993fd161: No bootstrap required, opened a new log
I20260619 16:41:13.675340 2723 ts_tablet_manager.cc:1397] T 8e5e8f385dbe405aa033952fd9c5b735 P 05e53be59c9f40c1806e9662993fd161: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 16:41:13.675679 2113 catalog_manager.cc:4841] TS 4ad5adf9d0e545eb8cccef1d682b37b9 (127.0.98.1:46343): tablet 0d82fa7d9fe34f0ca330900583fdb285 (table test-table [id=40cabd47627f4461bf295048296d3ce6]) successfully deleted
I20260619 16:41:13.675841 2113 catalog_manager.cc:4841] TS 3d782f18c34e43a3b39fc58ffdc8e1d9 (127.0.98.3:45745): tablet 0d82fa7d9fe34f0ca330900583fdb285 (table test-table [id=40cabd47627f4461bf295048296d3ce6]) successfully deleted
I20260619 16:41:13.675925 2723 raft_consensus.cc:348] T 8e5e8f385dbe405aa033952fd9c5b735 P 05e53be59c9f40c1806e9662993fd161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:13.676074 2723 raft_consensus.cc:374] T 8e5e8f385dbe405aa033952fd9c5b735 P 05e53be59c9f40c1806e9662993fd161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:13.676131 2723 raft_consensus.cc:729] T 8e5e8f385dbe405aa033952fd9c5b735 P 05e53be59c9f40c1806e9662993fd161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05e53be59c9f40c1806e9662993fd161, State: Initialized, Role: FOLLOWER
I20260619 16:41:13.676290 2723 consensus_queue.cc:260] T 8e5e8f385dbe405aa033952fd9c5b735 P 05e53be59c9f40c1806e9662993fd161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:13.676395 2724 tablet_bootstrap.cc:492] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9: No bootstrap required, opened a new log
I20260619 16:41:13.676504 2724 ts_tablet_manager.cc:1397] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260619 16:41:13.676589 2723 ts_tablet_manager.cc:1428] T 8e5e8f385dbe405aa033952fd9c5b735 P 05e53be59c9f40c1806e9662993fd161: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 16:41:13.676952 2724 raft_consensus.cc:348] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:13.677073 2724 raft_consensus.cc:374] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:13.677115 2724 raft_consensus.cc:729] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d782f18c34e43a3b39fc58ffdc8e1d9, State: Initialized, Role: FOLLOWER
I20260619 16:41:13.677253 2724 consensus_queue.cc:260] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:13.679814 2725 tablet_bootstrap.cc:492] T 8e5e8f385dbe405aa033952fd9c5b735 P 4ad5adf9d0e545eb8cccef1d682b37b9: Bootstrap starting.
I20260619 16:41:13.680174 2724 ts_tablet_manager.cc:1428] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 16:41:13.680949 2725 tablet_bootstrap.cc:654] T 8e5e8f385dbe405aa033952fd9c5b735 P 4ad5adf9d0e545eb8cccef1d682b37b9: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:13.682399 2725 tablet_bootstrap.cc:492] T 8e5e8f385dbe405aa033952fd9c5b735 P 4ad5adf9d0e545eb8cccef1d682b37b9: No bootstrap required, opened a new log
I20260619 16:41:13.682512 2725 ts_tablet_manager.cc:1397] T 8e5e8f385dbe405aa033952fd9c5b735 P 4ad5adf9d0e545eb8cccef1d682b37b9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:41:13.682904 2725 raft_consensus.cc:348] T 8e5e8f385dbe405aa033952fd9c5b735 P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:13.683058 2725 raft_consensus.cc:374] T 8e5e8f385dbe405aa033952fd9c5b735 P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:13.683094 2725 raft_consensus.cc:729] T 8e5e8f385dbe405aa033952fd9c5b735 P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ad5adf9d0e545eb8cccef1d682b37b9, State: Initialized, Role: FOLLOWER
I20260619 16:41:13.683238 2725 consensus_queue.cc:260] T 8e5e8f385dbe405aa033952fd9c5b735 P 4ad5adf9d0e545eb8cccef1d682b37b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:13.684021 2725 ts_tablet_manager.cc:1428] T 8e5e8f385dbe405aa033952fd9c5b735 P 4ad5adf9d0e545eb8cccef1d682b37b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:41:13.688730 2702 raft_consensus.cc:482] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:41:13.688875 2702 raft_consensus.cc:504] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:13.689461 2702 leader_election.cc:288] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ad5adf9d0e545eb8cccef1d682b37b9 (127.0.98.1:46343), 05e53be59c9f40c1806e9662993fd161 (127.0.98.2:40965)
I20260619 16:41:13.689867 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5e8f385dbe405aa033952fd9c5b735" candidate_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" is_pre_election: true
I20260619 16:41:13.690078 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5e8f385dbe405aa033952fd9c5b735" candidate_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05e53be59c9f40c1806e9662993fd161" is_pre_election: true
I20260619 16:41:13.690115 2284 raft_consensus.cc:2454] T 8e5e8f385dbe405aa033952fd9c5b735 P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d782f18c34e43a3b39fc58ffdc8e1d9 in term 0.
I20260619 16:41:13.690240 2424 raft_consensus.cc:2454] T 8e5e8f385dbe405aa033952fd9c5b735 P 05e53be59c9f40c1806e9662993fd161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d782f18c34e43a3b39fc58ffdc8e1d9 in term 0.
I20260619 16:41:13.690531 2497 leader_election.cc:302] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d782f18c34e43a3b39fc58ffdc8e1d9, 4ad5adf9d0e545eb8cccef1d682b37b9; no voters:
I20260619 16:41:13.690739 2702 raft_consensus.cc:2781] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:41:13.690814 2702 raft_consensus.cc:482] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:41:13.690871 2702 raft_consensus.cc:3037] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:13.692090 2702 raft_consensus.cc:504] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:13.692485 2702 leader_election.cc:288] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4ad5adf9d0e545eb8cccef1d682b37b9 (127.0.98.1:46343), 05e53be59c9f40c1806e9662993fd161 (127.0.98.2:40965)
I20260619 16:41:13.692898 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5e8f385dbe405aa033952fd9c5b735" candidate_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9"
I20260619 16:41:13.693051 2284 raft_consensus.cc:3037] T 8e5e8f385dbe405aa033952fd9c5b735 P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:13.693087 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5e8f385dbe405aa033952fd9c5b735" candidate_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05e53be59c9f40c1806e9662993fd161"
I20260619 16:41:13.693377 2424 raft_consensus.cc:3037] T 8e5e8f385dbe405aa033952fd9c5b735 P 05e53be59c9f40c1806e9662993fd161 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:13.694103 2284 raft_consensus.cc:2454] T 8e5e8f385dbe405aa033952fd9c5b735 P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d782f18c34e43a3b39fc58ffdc8e1d9 in term 1.
I20260619 16:41:13.694581 2497 leader_election.cc:302] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d782f18c34e43a3b39fc58ffdc8e1d9, 4ad5adf9d0e545eb8cccef1d682b37b9; no voters:
I20260619 16:41:13.694815 2702 raft_consensus.cc:2781] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:13.694921 2702 raft_consensus.cc:686] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 1 LEADER]: Becoming Leader. State: Replica: 3d782f18c34e43a3b39fc58ffdc8e1d9, State: Running, Role: LEADER
I20260619 16:41:13.695084 2702 consensus_queue.cc:237] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } }
I20260619 16:41:13.695556 2424 raft_consensus.cc:2454] T 8e5e8f385dbe405aa033952fd9c5b735 P 05e53be59c9f40c1806e9662993fd161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d782f18c34e43a3b39fc58ffdc8e1d9 in term 1.
I20260619 16:41:13.697341 2127 catalog_manager.cc:5495] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d782f18c34e43a3b39fc58ffdc8e1d9 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d782f18c34e43a3b39fc58ffdc8e1d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45745 } health_report { overall_health: HEALTHY } } }
I20260619 16:41:13.722023 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:13.717531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44796 (local address 127.0.98.3:45745)
0619 16:41:13.717902 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:13.717913 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:13.717932 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:41:13.717998 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:13.718001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:13.718065 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:13.718186 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:41:13.718543 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:13.719411 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:13.720602 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:13.720980 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:13.721094 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:13.721358 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:41:13.721466 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:41:13.721547 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:13.721751 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 16:41:13.721826 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":131,"threads_started":1}
I20260619 16:41:13.731704 2284 raft_consensus.cc:1261] T 8e5e8f385dbe405aa033952fd9c5b735 P 4ad5adf9d0e545eb8cccef1d682b37b9 [term 1 FOLLOWER]: Refusing update from remote peer 3d782f18c34e43a3b39fc58ffdc8e1d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:13.732219 2424 raft_consensus.cc:1261] T 8e5e8f385dbe405aa033952fd9c5b735 P 05e53be59c9f40c1806e9662993fd161 [term 1 FOLLOWER]: Refusing update from remote peer 3d782f18c34e43a3b39fc58ffdc8e1d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:13.732201 2702 consensus_queue.cc:1035] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ad5adf9d0e545eb8cccef1d682b37b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:13.732731 2707 consensus_queue.cc:1035] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05e53be59c9f40c1806e9662993fd161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:41:13.753211 2696 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 16:41:15.329774 2127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:41:15.330065 2127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:41:15.333956 2127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e74dc107d0148a4a928ae1bce521b71: Sending DeleteTablet for 3 replicas of tablet 8e5e8f385dbe405aa033952fd9c5b735
I20260619 16:41:15.334844 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e5e8f385dbe405aa033952fd9c5b735 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55712
I20260619 16:41:15.334926 2544 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e5e8f385dbe405aa033952fd9c5b735 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44770
I20260619 16:41:15.334930 2404 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e5e8f385dbe405aa033952fd9c5b735 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34622
I20260619 16:41:15.335284 2768 tablet_replica.cc:330] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9: stopping tablet replica
I20260619 16:41:15.335376 2769 tablet_replica.cc:330] T 8e5e8f385dbe405aa033952fd9c5b735 P 05e53be59c9f40c1806e9662993fd161: stopping tablet replica
I20260619 16:41:15.335945 2769 raft_consensus.cc:2229] T 8e5e8f385dbe405aa033952fd9c5b735 P 05e53be59c9f40c1806e9662993fd161 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:41:15.336197 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 2199
I20260619 16:41:15.336249 2769 raft_consensus.cc:2258] T 8e5e8f385dbe405aa033952fd9c5b735 P 05e53be59c9f40c1806e9662993fd161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:15.335564 2768 raft_consensus.cc:2229] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:41:15.339733 2768 raft_consensus.cc:2258] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:15.340253 2769 ts_tablet_manager.cc:1905] T 8e5e8f385dbe405aa033952fd9c5b735 P 05e53be59c9f40c1806e9662993fd161: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:15.340818 2768 ts_tablet_manager.cc:1905] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:15.344102 2768 ts_tablet_manager.cc:1918] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:41:15.344240 2768 log.cc:1196] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-2/wal/wals/8e5e8f385dbe405aa033952fd9c5b735
I20260619 16:41:15.344507 2769 ts_tablet_manager.cc:1918] T 8e5e8f385dbe405aa033952fd9c5b735 P 05e53be59c9f40c1806e9662993fd161: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:41:15.344578 2768 ts_tablet_manager.cc:1939] T 8e5e8f385dbe405aa033952fd9c5b735 P 3d782f18c34e43a3b39fc58ffdc8e1d9: Deleting consensus metadata
I20260619 16:41:15.344668 2769 log.cc:1196] T 8e5e8f385dbe405aa033952fd9c5b735 P 05e53be59c9f40c1806e9662993fd161: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887259574980-392-0/minicluster-data/ts-1/wal/wals/8e5e8f385dbe405aa033952fd9c5b735
I20260619 16:41:15.345065 2769 ts_tablet_manager.cc:1939] T 8e5e8f385dbe405aa033952fd9c5b735 P 05e53be59c9f40c1806e9662993fd161: Deleting consensus metadata
I20260619 16:41:15.346221 2113 catalog_manager.cc:4841] TS 3d782f18c34e43a3b39fc58ffdc8e1d9 (127.0.98.3:45745): tablet 8e5e8f385dbe405aa033952fd9c5b735 (table test-table [id=e82c2f33c28d41bdb2fd1642b2ab88d4]) successfully deleted
I20260619 16:41:15.346855 2112 catalog_manager.cc:4841] TS 05e53be59c9f40c1806e9662993fd161 (127.0.98.2:40965): tablet 8e5e8f385dbe405aa033952fd9c5b735 (table test-table [id=e82c2f33c28d41bdb2fd1642b2ab88d4]) successfully deleted
W20260619 16:41:15.346982 2113 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46343 (error 108)
I20260619 16:41:15.347398 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 2339
I20260619 16:41:15.353699 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:15.353098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46343 (local address 127.0.0.1:52862)
0619 16:41:15.353458 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:15.353572 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":159,"threads_started":1}
W20260619 16:41:15.353895 2113 catalog_manager.cc:4584] TS 4ad5adf9d0e545eb8cccef1d682b37b9 (127.0.98.1:46343): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8e5e8f385dbe405aa033952fd9c5b735: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46343: connect: Connection refused (error 111)
I20260619 16:41:15.354738 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 2479
I20260619 16:41:15.363300 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 2095
2026-06-19T16:41:15Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6798 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:41:15 dist-test-slave-03cd krb5kdc[2774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:41:15 dist-test-slave-03cd krb5kdc[2774](info): set up 2 sockets
Jun 19 16:41:15 dist-test-slave-03cd krb5kdc[2774](info): commencing operation
krb5kdc: starting...
W20260619 16:41:16.488312 392 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:41:16 dist-test-slave-03cd krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:41:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:41:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:41:16.644604 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36349
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:16.795621 2787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:16.796144 2787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:16.796228 2787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:16.801178 2787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:41:16.801337 2787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:16.801374 2787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:16.801414 2787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:41:16.801479 2787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:41:16.808514 2787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:16.810071 2787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:16.811425 2787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:16.822827 2793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:16.823076 2787 server_base.cc:1034] running on GCE node
W20260619 16:41:16.822914 2795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:16.822855 2792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:16.824225 2787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:16.825563 2787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:16.828943 2787 hybrid_clock.cc:648] HybridClock initialized: now 1781887276828900 us; error 64 us; skew 500 ppm
Jun 19 16:41:16 dist-test-slave-03cd krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:16.833112 2787 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:41:16.834434 2787 webserver.cc:460] Webserver started at http://127.0.98.62:35217/ using document root <none> and password file <none>
I20260619 16:41:16.835184 2787 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:16.835268 2787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:16.835546 2787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:16.838336 2787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "71e18a6fc8264388985a00b3376621cd"
format_stamp: "Formatted at 2026-06-19 16:41:16 on dist-test-slave-03cd"
server_key: "9aba4dd6bb1657d6febed1d2ee9b5e1b"
server_key_iv: "ded77b421a8e8ea63ff8586665215791"
server_key_version: "encryptionkey@0"
I20260619 16:41:16.839368 2787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "71e18a6fc8264388985a00b3376621cd"
format_stamp: "Formatted at 2026-06-19 16:41:16 on dist-test-slave-03cd"
server_key: "9aba4dd6bb1657d6febed1d2ee9b5e1b"
server_key_iv: "ded77b421a8e8ea63ff8586665215791"
server_key_version: "encryptionkey@0"
I20260619 16:41:16.844669 2787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260619 16:41:16.848481 2802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:16.850766 2787 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:41:16.851002 2787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "71e18a6fc8264388985a00b3376621cd"
format_stamp: "Formatted at 2026-06-19 16:41:16 on dist-test-slave-03cd"
server_key: "9aba4dd6bb1657d6febed1d2ee9b5e1b"
server_key_iv: "ded77b421a8e8ea63ff8586665215791"
server_key_version: "encryptionkey@0"
I20260619 16:41:16.851176 2787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:16.870312 2787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:16.874606 2787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:16.874819 2787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:16.886251 2787 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36349
I20260619 16:41:16.886376 2854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36349 every 8 connection(s)
I20260619 16:41:16.887622 2787 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:41:16.891980 2855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:16.894563 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 2787
I20260619 16:41:16.894723 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:41:16.895061 392 external_mini_cluster.cc:1428] Setting key b09067fc913c7dfcd494fbf8c4b17431
I20260619 16:41:16.901968 2855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71e18a6fc8264388985a00b3376621cd: Bootstrap starting.
I20260619 16:41:16.906544 2855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71e18a6fc8264388985a00b3376621cd: Neither blocks nor log segments found. Creating new log.
Jun 19 16:41:16 dist-test-slave-03cd krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:16.908203 2855 log.cc:824] T 00000000000000000000000000000000 P 71e18a6fc8264388985a00b3376621cd: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:16.913048 2855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71e18a6fc8264388985a00b3376621cd: No bootstrap required, opened a new log
I20260619 16:41:16.915979 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:16.897282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40556 (local address 127.0.98.62:36349)
0619 16:41:16.898082 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:16.898104 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:16.898155 (+ 51us) server_negotiation.cc:407] Connection header received
0619 16:41:16.899063 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:16.899088 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:16.899569 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:16.900070 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:16.901001 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:16.902341 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:16.903310 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:16.904112 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:16.904352 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:16.908827 (+ 4475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:16.908838 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:16.908870 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:16.908919 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:16.911470 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:16.912157 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:16.912164 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:16.912170 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:16.912313 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:16.912847 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:16.912854 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:16.912859 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:16.913598 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:16.913875 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:16.914185 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 16:41:16.914638 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":226,"threads_started":1}
I20260619 16:41:16.919054 2855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71e18a6fc8264388985a00b3376621cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e18a6fc8264388985a00b3376621cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36349 } }
I20260619 16:41:16.919919 2855 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71e18a6fc8264388985a00b3376621cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:16.920037 2855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71e18a6fc8264388985a00b3376621cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e18a6fc8264388985a00b3376621cd, State: Initialized, Role: FOLLOWER
I20260619 16:41:16.920733 2855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71e18a6fc8264388985a00b3376621cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e18a6fc8264388985a00b3376621cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36349 } }
I20260619 16:41:16.921104 2855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71e18a6fc8264388985a00b3376621cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:41:16.921187 2855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71e18a6fc8264388985a00b3376621cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:41:16.921294 2855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71e18a6fc8264388985a00b3376621cd [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:16.923120 2855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71e18a6fc8264388985a00b3376621cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e18a6fc8264388985a00b3376621cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36349 } }
I20260619 16:41:16.923651 2855 leader_election.cc:302] T 00000000000000000000000000000000 P 71e18a6fc8264388985a00b3376621cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71e18a6fc8264388985a00b3376621cd; no voters:
I20260619 16:41:16.924335 2855 leader_election.cc:288] T 00000000000000000000000000000000 P 71e18a6fc8264388985a00b3376621cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:41:16.924559 2860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71e18a6fc8264388985a00b3376621cd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:16.925298 2860 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71e18a6fc8264388985a00b3376621cd [term 1 LEADER]: Becoming Leader. State: Replica: 71e18a6fc8264388985a00b3376621cd, State: Running, Role: LEADER
I20260619 16:41:16.925835 2860 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71e18a6fc8264388985a00b3376621cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e18a6fc8264388985a00b3376621cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36349 } }
I20260619 16:41:16.926167 2855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71e18a6fc8264388985a00b3376621cd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:41:16.927951 2861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71e18a6fc8264388985a00b3376621cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71e18a6fc8264388985a00b3376621cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e18a6fc8264388985a00b3376621cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36349 } } }
I20260619 16:41:16.927951 2862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71e18a6fc8264388985a00b3376621cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71e18a6fc8264388985a00b3376621cd. Latest consensus state: current_term: 1 leader_uuid: "71e18a6fc8264388985a00b3376621cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e18a6fc8264388985a00b3376621cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36349 } } }
I20260619 16:41:16.928155 2862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71e18a6fc8264388985a00b3376621cd [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:16.928242 2861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71e18a6fc8264388985a00b3376621cd [sys.catalog]: This master's current role is: LEADER
W20260619 16:41:16.932906 2875 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 71e18a6fc8264388985a00b3376621cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:41:16.933056 2875 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:41:16.933246 2873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:41:16.934337 2873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:41:16.942126 2873 catalog_manager.cc:1347] Generated new cluster ID: 552062b0adf946399bccb71a98241da5
I20260619 16:41:16.942348 2873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:41:16.958879 2873 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:41:16.960354 2873 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:41:16.968340 2873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71e18a6fc8264388985a00b3376621cd: Generated new TSK 0
I20260619 16:41:16.969238 2873 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:41:17.012395 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36349
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:17.157923 2881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:17.158255 2881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:17.158306 2881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:17.162719 2881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:17.162812 2881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:17.162878 2881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:41:17.169760 2881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:17.171033 2881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:17.172228 2881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:17.184224 2889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:17.184346 2887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:17.184224 2886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:17.184593 2881 server_base.cc:1034] running on GCE node
I20260619 16:41:17.185225 2881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:17.185961 2881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:17.187247 2881 hybrid_clock.cc:648] HybridClock initialized: now 1781887277187213 us; error 54 us; skew 500 ppm
Jun 19 16:41:17 dist-test-slave-03cd krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:17.191051 2881 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:41:17.192301 2881 webserver.cc:460] Webserver started at http://127.0.98.1:43215/ using document root <none> and password file <none>
I20260619 16:41:17.193058 2881 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:17.193212 2881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:17.193470 2881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:17.195868 2881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/data/instance:
uuid: "c1ba28e2476d41c68f03f5832f07963a"
format_stamp: "Formatted at 2026-06-19 16:41:17 on dist-test-slave-03cd"
server_key: "3e567b2c0d5fa2bf7481e155c9c0460e"
server_key_iv: "4f271671eed258805e64676ccd73959f"
server_key_version: "encryptionkey@0"
I20260619 16:41:17.196507 2881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c1ba28e2476d41c68f03f5832f07963a"
format_stamp: "Formatted at 2026-06-19 16:41:17 on dist-test-slave-03cd"
server_key: "3e567b2c0d5fa2bf7481e155c9c0460e"
server_key_iv: "4f271671eed258805e64676ccd73959f"
server_key_version: "encryptionkey@0"
I20260619 16:41:17.201290 2881 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:41:17.204794 2896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:17.206851 2881 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:41:17.207018 2881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "c1ba28e2476d41c68f03f5832f07963a"
format_stamp: "Formatted at 2026-06-19 16:41:17 on dist-test-slave-03cd"
server_key: "3e567b2c0d5fa2bf7481e155c9c0460e"
server_key_iv: "4f271671eed258805e64676ccd73959f"
server_key_version: "encryptionkey@0"
I20260619 16:41:17.207151 2881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:17.234781 2881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:17.238742 2881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:17.238921 2881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:17.239492 2881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:17.240477 2881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:17.240541 2881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:17.240628 2881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:17.240664 2881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:17.251359 2881 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43731
I20260619 16:41:17.251454 3009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43731 every 8 connection(s)
I20260619 16:41:17.252481 2881 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:41:17.261497 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 2881
I20260619 16:41:17.261628 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:41:17.261921 392 external_mini_cluster.cc:1428] Setting key 147c5106277588955eabcb7fe3ea6c24
Jun 19 16:41:17 dist-test-slave-03cd krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:17.273283 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:17.255379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33775 (local address 127.0.98.62:36349)
0619 16:41:17.255586 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:17.255591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:17.256966 (+ 1375us) server_negotiation.cc:407] Connection header received
0619 16:41:17.258552 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:17.258555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:17.258617 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:17.258726 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:17.260186 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:17.260741 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:17.262399 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:17.262947 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:17.263145 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:17.266748 (+ 3603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:17.266764 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:17.266768 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:17.266822 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:17.269308 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:17.270328 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:17.270335 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:17.270337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:17.270543 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:17.271297 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:17.271305 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:17.271310 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:17.271799 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:17.271941 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:17.272835 (+ 894us) server_negotiation.cc:299] Negotiation successful
0619 16:41:17.273064 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 16:41:17.274369 3012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:17.256011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36349 (local address 127.0.98.1:33775)
0619 16:41:17.256719 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:17.256806 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:17.258290 (+ 1484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:17.258936 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:17.258944 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:17.259439 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:17.260006 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:17.260022 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:17.260882 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:17.260888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:17.262158 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:17.262171 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:17.263211 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:17.263216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:17.263496 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:17.264453 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:17.264475 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:17.266491 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:17.269494 (+ 3003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:17.269504 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:17.269529 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:17.270096 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:17.270709 (+ 613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:17.270713 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:17.270718 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:17.271085 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:17.271965 (+ 880us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:17.271975 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:17.272492 (+ 517us) client_negotiation.cc:769] Sending connection context
0619 16:41:17.272755 (+ 263us) client_negotiation.cc:240] Negotiation successful
0619 16:41:17.273215 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":195,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:41:17.275980 3010 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36349
I20260619 16:41:17.276283 3010 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:17.276955 3010 heartbeater.cc:511] Master 127.0.98.62:36349 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 16:41:17.279096 2819 ts_manager.cc:194] Registered new tserver with Master: c1ba28e2476d41c68f03f5832f07963a (127.0.98.1:43731)
I20260619 16:41:17.280567 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:41:17.295578 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36349
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:17.425115 3015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:17.425500 3015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:17.425549 3015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:17.430496 3015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:17.430595 3015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:17.430665 3015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:41:17.436820 3015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:17.438079 3015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:17.439306 3015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:17.451086 3020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:17.451154 3021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:17.451112 3023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:17.451516 3015 server_base.cc:1034] running on GCE node
I20260619 16:41:17.452167 3015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:17.452970 3015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:17.454214 3015 hybrid_clock.cc:648] HybridClock initialized: now 1781887277454197 us; error 39 us; skew 500 ppm
Jun 19 16:41:17 dist-test-slave-03cd krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:17.457912 3015 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:41:17.459165 3015 webserver.cc:460] Webserver started at http://127.0.98.2:43017/ using document root <none> and password file <none>
I20260619 16:41:17.459976 3015 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:17.460059 3015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:17.460279 3015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:17.462908 3015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/data/instance:
uuid: "69b16569fa444d148817e303e2f30490"
format_stamp: "Formatted at 2026-06-19 16:41:17 on dist-test-slave-03cd"
server_key: "6beffe5fb47b005dd6b847a9f7ee28f4"
server_key_iv: "50d36179bd59e92cd485e666bb1d519f"
server_key_version: "encryptionkey@0"
I20260619 16:41:17.463595 3015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/wal/instance:
uuid: "69b16569fa444d148817e303e2f30490"
format_stamp: "Formatted at 2026-06-19 16:41:17 on dist-test-slave-03cd"
server_key: "6beffe5fb47b005dd6b847a9f7ee28f4"
server_key_iv: "50d36179bd59e92cd485e666bb1d519f"
server_key_version: "encryptionkey@0"
I20260619 16:41:17.469504 3015 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260619 16:41:17.473202 3030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:17.475473 3015 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:41:17.475735 3015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "69b16569fa444d148817e303e2f30490"
format_stamp: "Formatted at 2026-06-19 16:41:17 on dist-test-slave-03cd"
server_key: "6beffe5fb47b005dd6b847a9f7ee28f4"
server_key_iv: "50d36179bd59e92cd485e666bb1d519f"
server_key_version: "encryptionkey@0"
I20260619 16:41:17.475917 3015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:17.496901 3015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:17.501906 3015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:17.502218 3015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:17.503144 3015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:17.504655 3015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:17.504747 3015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:17.504863 3015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:17.504904 3015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:17.518386 3015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33445
I20260619 16:41:17.518407 3143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33445 every 8 connection(s)
I20260619 16:41:17.520129 3015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:41:17.524605 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 3015
I20260619 16:41:17.524761 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:41:17.525086 392 external_mini_cluster.cc:1428] Setting key 41c5d4759e512a77fc926d83ddc402de
Jun 19 16:41:17 dist-test-slave-03cd krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:41:17.544303 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:17.523085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49153 (local address 127.0.98.62:36349)
0619 16:41:17.523375 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:17.523379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:17.524927 (+ 1548us) server_negotiation.cc:407] Connection header received
0619 16:41:17.527997 (+ 3070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:17.528000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:17.528055 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:17.528125 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:17.531075 (+ 2950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:17.531847 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:17.534167 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:17.534581 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:17.534745 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:17.539132 (+ 4387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:17.539137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:17.539139 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:17.539175 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:17.540887 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:17.541821 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:17.541825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:17.541827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:17.541935 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:17.542697 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:17.542701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:17.542703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:17.542965 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:17.543088 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:17.543819 (+ 731us) server_negotiation.cc:299] Negotiation successful
0619 16:41:17.544101 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260619 16:41:17.545872 3146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:17.523735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36349 (local address 127.0.98.2:49153)
0619 16:41:17.524577 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:17.524646 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:17.527629 (+ 2983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:17.528437 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:17.528446 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:17.528992 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:17.529970 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:17.530001 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:17.532024 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:17.532033 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:17.533969 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:17.533997 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:17.535047 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:17.535054 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:17.535330 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:17.536365 (+ 1035us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:17.536390 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:17.538825 (+ 2435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:17.541076 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:17.541086 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:17.541111 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:17.541646 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:17.542042 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:17.542046 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:17.542048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:17.542257 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:17.543090 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:17.543101 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:17.543475 (+ 374us) client_negotiation.cc:769] Sending connection context
0619 16:41:17.543777 (+ 302us) client_negotiation.cc:240] Negotiation successful
0619 16:41:17.544407 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":214,"threads_started":1}
I20260619 16:41:17.547869 3144 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36349
I20260619 16:41:17.548334 3144 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:17.549074 3144 heartbeater.cc:511] Master 127.0.98.62:36349 requested a full tablet report, sending...
I20260619 16:41:17.550395 2819 ts_manager.cc:194] Registered new tserver with Master: 69b16569fa444d148817e303e2f30490 (127.0.98.2:33445)
I20260619 16:41:17.551122 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260619 16:41:17.975127 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.444s user 0.000s sys 0.006s
I20260619 16:41:17.975904 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36349
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:18.131583 3149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:18.132045 3149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:18.132105 3149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:18.137760 3149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:18.137868 3149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:18.137943 3149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:41:18.144760 3149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:18.146286 3149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:18.147682 3149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:18.160187 3157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:18.160198 3155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:18.160195 3154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:18.160570 3149 server_base.cc:1034] running on GCE node
I20260619 16:41:18.161165 3149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:18.161922 3149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:18.163120 3149 hybrid_clock.cc:648] HybridClock initialized: now 1781887278163087 us; error 58 us; skew 500 ppm
Jun 19 16:41:18 dist-test-slave-03cd krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:18.166802 3149 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:41:18.168078 3149 webserver.cc:460] Webserver started at http://127.0.98.3:35025/ using document root <none> and password file <none>
I20260619 16:41:18.168808 3149 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:18.168886 3149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:18.169137 3149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:18.171643 3149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/data/instance:
uuid: "f004701cb38240a9b56121d32697f467"
format_stamp: "Formatted at 2026-06-19 16:41:18 on dist-test-slave-03cd"
server_key: "f721a332618ac47ed58a16e6765c3ad3"
server_key_iv: "8d10f23ffa003652736f26d3e13c5187"
server_key_version: "encryptionkey@0"
I20260619 16:41:18.172389 3149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f004701cb38240a9b56121d32697f467"
format_stamp: "Formatted at 2026-06-19 16:41:18 on dist-test-slave-03cd"
server_key: "f721a332618ac47ed58a16e6765c3ad3"
server_key_iv: "8d10f23ffa003652736f26d3e13c5187"
server_key_version: "encryptionkey@0"
I20260619 16:41:18.177423 3149 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 16:41:18.181253 3164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:18.183755 3149 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20260619 16:41:18.183935 3149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "f004701cb38240a9b56121d32697f467"
format_stamp: "Formatted at 2026-06-19 16:41:18 on dist-test-slave-03cd"
server_key: "f721a332618ac47ed58a16e6765c3ad3"
server_key_iv: "8d10f23ffa003652736f26d3e13c5187"
server_key_version: "encryptionkey@0"
I20260619 16:41:18.184121 3149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:18.205582 3149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:18.210877 3149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:18.211170 3149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:18.212234 3149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:18.213551 3149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:18.213649 3149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:18.213737 3149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:18.213791 3149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:18.227380 3149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34263
I20260619 16:41:18.227473 3277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34263 every 8 connection(s)
I20260619 16:41:18.228878 3149 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:41:18.236807 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 3149
I20260619 16:41:18.236948 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:41:18.237313 392 external_mini_cluster.cc:1428] Setting key dd0b89184ba0ee54ffa03ccc5c7610f9
Jun 19 16:41:18 dist-test-slave-03cd krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:18.252530 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:18.231508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37843 (local address 127.0.98.62:36349)
0619 16:41:18.231826 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:18.231830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:18.233083 (+ 1253us) server_negotiation.cc:407] Connection header received
0619 16:41:18.235375 (+ 2292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:18.235379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:18.235455 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:18.235558 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:18.237526 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:18.238328 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:18.240883 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:18.241326 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:18.241487 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:18.246394 (+ 4907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:18.246403 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:18.246410 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:18.246462 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:18.248974 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:18.250027 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:18.250031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:18.250032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:18.250167 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:18.250738 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:18.250741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:18.250743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:18.250972 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:18.251090 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:18.252104 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0619 16:41:18.252301 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":120,"threads_started":1}
I20260619 16:41:18.253934 3281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:18.232130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36349 (local address 127.0.98.3:37843)
0619 16:41:18.232859 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:18.232925 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:18.235045 (+ 2120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:18.235838 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:18.235846 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:18.236626 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:18.237307 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:18.237323 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:18.238588 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:18.238596 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:18.240678 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:18.240701 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:18.241596 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:18.241601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:18.241944 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:18.242940 (+ 996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:18.242967 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:18.245930 (+ 2963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:18.249202 (+ 3272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:18.249216 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:18.249247 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:18.249832 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:18.250330 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:18.250334 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:18.250336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:18.250564 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:18.251217 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:18.251226 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:18.251737 (+ 511us) client_negotiation.cc:769] Sending connection context
0619 16:41:18.252004 (+ 267us) client_negotiation.cc:240] Negotiation successful
0619 16:41:18.252515 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":196,"threads_started":1}
I20260619 16:41:18.255367 3278 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36349
I20260619 16:41:18.255736 3278 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:18.256462 3278 heartbeater.cc:511] Master 127.0.98.62:36349 requested a full tablet report, sending...
I20260619 16:41:18.257798 2819 ts_manager.cc:194] Registered new tserver with Master: f004701cb38240a9b56121d32697f467 (127.0.98.3:34263)
I20260619 16:41:18.258606 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37843
I20260619 16:41:18.266577 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 16:41:18.283632 3280 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:18.281452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40560 (local address 127.0.98.62:36349)
0619 16:41:18.281670 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:18.281675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:18.281976 (+ 301us) server_negotiation.cc:407] Connection header received
0619 16:41:18.282220 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:18.282224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:18.282279 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:18.282387 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:18.283461 (+ 1074us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40560: BlockingRecv error: recv got EOF from 127.0.0.1:40560 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:41:18.283650 3010 heartbeater.cc:503] Master 127.0.98.62:36349 was elected leader, sending a full tablet report...
I20260619 16:41:18.286022 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 2881
I20260619 16:41:18.293704 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 3015
I20260619 16:41:18.301963 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 3149
I20260619 16:41:18.310298 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 2787
2026-06-19T16:41:18Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2944 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:41:18 dist-test-slave-03cd krb5kdc[3291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:41:18 dist-test-slave-03cd krb5kdc[3291](info): set up 2 sockets
Jun 19 16:41:18 dist-test-slave-03cd krb5kdc[3291](info): commencing operation
krb5kdc: starting...
W20260619 16:41:19.424219 392 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:41:19 dist-test-slave-03cd krb5kdc[3291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:41:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:41:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:41:19.576606 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38727
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:19.710644 3304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:19.710987 3304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:19.711035 3304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:19.716189 3304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:41:19.716275 3304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:19.716297 3304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:19.716317 3304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:41:19.716336 3304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:41:19.723055 3304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:19.724334 3304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:19.725371 3304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:19.736140 3310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:19.736099 3312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:19.736088 3309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:19.736728 3304 server_base.cc:1034] running on GCE node
I20260619 16:41:19.737396 3304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:19.738634 3304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:19.739835 3304 hybrid_clock.cc:648] HybridClock initialized: now 1781887279739807 us; error 50 us; skew 500 ppm
Jun 19 16:41:19 dist-test-slave-03cd krb5kdc[3291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:19.742980 3304 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:41:19.744282 3304 webserver.cc:460] Webserver started at http://127.0.98.62:35439/ using document root <none> and password file <none>
I20260619 16:41:19.744937 3304 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:19.745041 3304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:19.745297 3304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:19.747562 3304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "c7ed206f6f2a4b3c963cc5e967ce63de"
format_stamp: "Formatted at 2026-06-19 16:41:19 on dist-test-slave-03cd"
server_key: "a099ab98c481e0dfb5c82b5f1b313298"
server_key_iv: "3e12b6769402897e3d958afdd7411927"
server_key_version: "encryptionkey@0"
I20260619 16:41:19.748293 3304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "c7ed206f6f2a4b3c963cc5e967ce63de"
format_stamp: "Formatted at 2026-06-19 16:41:19 on dist-test-slave-03cd"
server_key: "a099ab98c481e0dfb5c82b5f1b313298"
server_key_iv: "3e12b6769402897e3d958afdd7411927"
server_key_version: "encryptionkey@0"
I20260619 16:41:19.753033 3304 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:41:19.756023 3319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:19.757541 3304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:41:19.757745 3304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "c7ed206f6f2a4b3c963cc5e967ce63de"
format_stamp: "Formatted at 2026-06-19 16:41:19 on dist-test-slave-03cd"
server_key: "a099ab98c481e0dfb5c82b5f1b313298"
server_key_iv: "3e12b6769402897e3d958afdd7411927"
server_key_version: "encryptionkey@0"
I20260619 16:41:19.757896 3304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:19.772096 3304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:19.776067 3304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:19.776322 3304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:19.783901 3304 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38727
I20260619 16:41:19.784216 3371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38727 every 8 connection(s)
I20260619 16:41:19.784996 3304 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:41:19.788406 3372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:19.794694 3372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7ed206f6f2a4b3c963cc5e967ce63de: Bootstrap starting.
I20260619 16:41:19.794839 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 3304
I20260619 16:41:19.795079 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:41:19.795392 392 external_mini_cluster.cc:1428] Setting key 8ab381b2eeabcaf59fe20175311b18b2
I20260619 16:41:19.797850 3372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7ed206f6f2a4b3c963cc5e967ce63de: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:19.798659 3372 log.cc:824] T 00000000000000000000000000000000 P c7ed206f6f2a4b3c963cc5e967ce63de: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:19.801383 3372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7ed206f6f2a4b3c963cc5e967ce63de: No bootstrap required, opened a new log
Jun 19 16:41:19 dist-test-slave-03cd krb5kdc[3291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:19.806526 3372 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7ed206f6f2a4b3c963cc5e967ce63de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7ed206f6f2a4b3c963cc5e967ce63de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38727 } }
I20260619 16:41:19.806843 3372 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7ed206f6f2a4b3c963cc5e967ce63de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:19.806924 3372 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7ed206f6f2a4b3c963cc5e967ce63de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7ed206f6f2a4b3c963cc5e967ce63de, State: Initialized, Role: FOLLOWER
I20260619 16:41:19.807511 3372 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7ed206f6f2a4b3c963cc5e967ce63de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7ed206f6f2a4b3c963cc5e967ce63de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38727 } }
I20260619 16:41:19.807698 3372 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7ed206f6f2a4b3c963cc5e967ce63de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:41:19.807776 3372 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7ed206f6f2a4b3c963cc5e967ce63de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:41:19.807888 3372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7ed206f6f2a4b3c963cc5e967ce63de [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:19.809176 3372 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7ed206f6f2a4b3c963cc5e967ce63de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7ed206f6f2a4b3c963cc5e967ce63de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38727 } }
I20260619 16:41:19.809631 3372 leader_election.cc:302] T 00000000000000000000000000000000 P c7ed206f6f2a4b3c963cc5e967ce63de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7ed206f6f2a4b3c963cc5e967ce63de; no voters:
I20260619 16:41:19.810019 3372 leader_election.cc:288] T 00000000000000000000000000000000 P c7ed206f6f2a4b3c963cc5e967ce63de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:41:19.810148 3377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7ed206f6f2a4b3c963cc5e967ce63de [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:19.810370 3377 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7ed206f6f2a4b3c963cc5e967ce63de [term 1 LEADER]: Becoming Leader. State: Replica: c7ed206f6f2a4b3c963cc5e967ce63de, State: Running, Role: LEADER
I20260619 16:41:19.810717 3377 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7ed206f6f2a4b3c963cc5e967ce63de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7ed206f6f2a4b3c963cc5e967ce63de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38727 } }
I20260619 16:41:19.811453 3372 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7ed206f6f2a4b3c963cc5e967ce63de [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:41:19.812062 3379 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7ed206f6f2a4b3c963cc5e967ce63de [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7ed206f6f2a4b3c963cc5e967ce63de. Latest consensus state: current_term: 1 leader_uuid: "c7ed206f6f2a4b3c963cc5e967ce63de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7ed206f6f2a4b3c963cc5e967ce63de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38727 } } }
I20260619 16:41:19.812184 3379 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7ed206f6f2a4b3c963cc5e967ce63de [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:19.812839 3378 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7ed206f6f2a4b3c963cc5e967ce63de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7ed206f6f2a4b3c963cc5e967ce63de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7ed206f6f2a4b3c963cc5e967ce63de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38727 } } }
I20260619 16:41:19.812937 3378 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7ed206f6f2a4b3c963cc5e967ce63de [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:19.812922 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:19.797012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48804 (local address 127.0.98.62:38727)
0619 16:41:19.797543 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:19.797564 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:19.797601 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:41:19.798462 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:19.798487 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:19.798891 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:19.799422 (+ 531us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:19.800667 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:19.801789 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:19.803228 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:19.803881 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:19.804036 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:19.807100 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:19.807107 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:19.807125 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:19.807163 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:19.809390 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:19.809874 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:19.809880 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:19.809886 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:19.810037 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:19.810388 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:19.810391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:19.810392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:19.810921 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:19.811129 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:19.811439 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 16:41:19.811784 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":151,"threads_started":1}
I20260619 16:41:19.816470 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 16:41:19.817366 3392 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c7ed206f6f2a4b3c963cc5e967ce63de: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:41:19.817445 3392 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:41:19.817515 3383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:41:19.818338 3383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:41:19.824700 3383 catalog_manager.cc:1347] Generated new cluster ID: deb1bde47d3e47319055f4029be27071
I20260619 16:41:19.824781 3383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:41:19.836709 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:19.831711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48820 (local address 127.0.98.62:38727)
0619 16:41:19.831848 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:19.831853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:19.832021 (+ 168us) server_negotiation.cc:407] Connection header received
0619 16:41:19.832194 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:19.832196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:19.832242 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:19.832330 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:19.832890 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:19.833428 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:19.834302 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:19.834576 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:19.834718 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:19.835638 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:19.835642 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:19.835645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:41:19.836214 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:19.836248 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:41:19.836264 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:19.836371 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:19.836479 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 16:41:19.836549 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 16:41:19.848388 3383 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:41:19.849783 3383 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:41:19.856268 3383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7ed206f6f2a4b3c963cc5e967ce63de: Generated new TSK 0
I20260619 16:41:19.857007 3383 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:41:19.862795 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 3304
2026-06-19T16:41:19Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1551 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:41:19 dist-test-slave-03cd krb5kdc[3405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:41:19 dist-test-slave-03cd krb5kdc[3405](info): set up 2 sockets
Jun 19 16:41:19 dist-test-slave-03cd krb5kdc[3405](info): commencing operation
krb5kdc: starting...
W20260619 16:41:20.944272 392 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:41:21 dist-test-slave-03cd krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:41:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:41:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:41:21.096710 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43225
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43225
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:21.217018 3418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:21.217401 3418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:21.217495 3418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:21.222864 3418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:41:21.223001 3418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:21.223042 3418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:21.223083 3418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:41:21.223124 3418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:41:21.229578 3418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:21.230801 3418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:21.231894 3418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:21.241473 3424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:21.241474 3423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:21.241483 3426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:21.241978 3418 server_base.cc:1034] running on GCE node
I20260619 16:41:21.242808 3418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:21.244462 3418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:21.245697 3418 hybrid_clock.cc:648] HybridClock initialized: now 1781887281245641 us; error 85 us; skew 500 ppm
Jun 19 16:41:21 dist-test-slave-03cd krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:21.253796 3418 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:41:21.255156 3418 webserver.cc:460] Webserver started at http://127.0.98.62:39889/ using document root <none> and password file <none>
I20260619 16:41:21.255978 3418 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:21.256078 3418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:21.256349 3418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:21.258978 3418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "6443ed58709e459eb675294507a5b5db"
format_stamp: "Formatted at 2026-06-19 16:41:21 on dist-test-slave-03cd"
server_key: "747089c643729bf1a4860a80df125f28"
server_key_iv: "90f1c96ceaec174e022a2dbd48f5fd00"
server_key_version: "encryptionkey@0"
I20260619 16:41:21.259788 3418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "6443ed58709e459eb675294507a5b5db"
format_stamp: "Formatted at 2026-06-19 16:41:21 on dist-test-slave-03cd"
server_key: "747089c643729bf1a4860a80df125f28"
server_key_iv: "90f1c96ceaec174e022a2dbd48f5fd00"
server_key_version: "encryptionkey@0"
I20260619 16:41:21.266605 3418 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 16:41:21.270136 3433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:21.271847 3418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:41:21.272056 3418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "6443ed58709e459eb675294507a5b5db"
format_stamp: "Formatted at 2026-06-19 16:41:21 on dist-test-slave-03cd"
server_key: "747089c643729bf1a4860a80df125f28"
server_key_iv: "90f1c96ceaec174e022a2dbd48f5fd00"
server_key_version: "encryptionkey@0"
I20260619 16:41:21.272207 3418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:21.287644 3418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:21.291476 3418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:21.291777 3418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:21.300168 3418 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43225
I20260619 16:41:21.300647 3485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43225 every 8 connection(s)
I20260619 16:41:21.301399 3418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:41:21.305553 3486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:21.305538 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 3418
I20260619 16:41:21.305691 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:41:21.306051 392 external_mini_cluster.cc:1428] Setting key 5e5aa3ec6958b1db8eac20aaf5387502
I20260619 16:41:21.314062 3486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6443ed58709e459eb675294507a5b5db: Bootstrap starting.
Jun 19 16:41:21 dist-test-slave-03cd krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:21.317337 3486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6443ed58709e459eb675294507a5b5db: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:21.318286 3486 log.cc:824] T 00000000000000000000000000000000 P 6443ed58709e459eb675294507a5b5db: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:21.321465 3486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6443ed58709e459eb675294507a5b5db: No bootstrap required, opened a new log
I20260619 16:41:21.324527 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:21.307994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59606 (local address 127.0.98.62:43225)
0619 16:41:21.308687 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:21.308706 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:21.308755 (+ 49us) server_negotiation.cc:407] Connection header received
0619 16:41:21.309657 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:21.309676 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:21.310114 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:21.310532 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:21.311431 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:21.312766 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:21.313746 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:21.314367 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:21.314549 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:21.317381 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:21.317388 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:21.317406 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:21.317442 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:21.319708 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:21.320301 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:21.320308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:21.320315 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:21.320494 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:21.320861 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:21.320865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:21.320867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:21.321472 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:21.321709 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:21.322123 (+ 414us) server_negotiation.cc:299] Negotiation successful
0619 16:41:21.322669 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":203,"threads_started":1}
I20260619 16:41:21.326885 3486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6443ed58709e459eb675294507a5b5db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6443ed58709e459eb675294507a5b5db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43225 } }
I20260619 16:41:21.327229 3486 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6443ed58709e459eb675294507a5b5db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:21.327267 3486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6443ed58709e459eb675294507a5b5db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6443ed58709e459eb675294507a5b5db, State: Initialized, Role: FOLLOWER
I20260619 16:41:21.327994 3486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6443ed58709e459eb675294507a5b5db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6443ed58709e459eb675294507a5b5db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43225 } }
I20260619 16:41:21.328204 3486 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6443ed58709e459eb675294507a5b5db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:41:21.328276 3486 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6443ed58709e459eb675294507a5b5db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:41:21.328387 3486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6443ed58709e459eb675294507a5b5db [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:21.329689 3486 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6443ed58709e459eb675294507a5b5db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6443ed58709e459eb675294507a5b5db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43225 } }
I20260619 16:41:21.330165 3486 leader_election.cc:302] T 00000000000000000000000000000000 P 6443ed58709e459eb675294507a5b5db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6443ed58709e459eb675294507a5b5db; no voters:
I20260619 16:41:21.330644 3486 leader_election.cc:288] T 00000000000000000000000000000000 P 6443ed58709e459eb675294507a5b5db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:41:21.330935 3491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6443ed58709e459eb675294507a5b5db [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:21.331396 3491 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6443ed58709e459eb675294507a5b5db [term 1 LEADER]: Becoming Leader. State: Replica: 6443ed58709e459eb675294507a5b5db, State: Running, Role: LEADER
I20260619 16:41:21.331933 3491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6443ed58709e459eb675294507a5b5db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6443ed58709e459eb675294507a5b5db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43225 } }
I20260619 16:41:21.332468 3486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6443ed58709e459eb675294507a5b5db [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:41:21.334206 3493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6443ed58709e459eb675294507a5b5db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6443ed58709e459eb675294507a5b5db. Latest consensus state: current_term: 1 leader_uuid: "6443ed58709e459eb675294507a5b5db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6443ed58709e459eb675294507a5b5db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43225 } } }
I20260619 16:41:21.334379 3493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6443ed58709e459eb675294507a5b5db [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:21.334810 3492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6443ed58709e459eb675294507a5b5db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6443ed58709e459eb675294507a5b5db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6443ed58709e459eb675294507a5b5db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43225 } } }
I20260619 16:41:21.334880 3502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:41:21.335637 3492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6443ed58709e459eb675294507a5b5db [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:21.338498 3502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:41:21.346192 3502 catalog_manager.cc:1347] Generated new cluster ID: 4d4d227de7a44186a347316a0564efa2
I20260619 16:41:21.346318 3502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:41:21.362259 3502 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:41:21.364319 3502 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:41:21.372782 3502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6443ed58709e459eb675294507a5b5db: Generated new TSK 0
I20260619 16:41:21.373862 3502 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:41:21.420058 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43225
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:21.567487 3512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:21.567804 3512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:21.567849 3512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:21.572067 3512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:21.572150 3512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:21.572212 3512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:41:21.577972 3512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:21.579187 3512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:21.580295 3512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:21.590655 3520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:21.590682 3517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:21.590758 3512 server_base.cc:1034] running on GCE node
W20260619 16:41:21.590698 3518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:21.591395 3512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:21.592207 3512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:21.593431 3512 hybrid_clock.cc:648] HybridClock initialized: now 1781887281593401 us; error 52 us; skew 500 ppm
Jun 19 16:41:21 dist-test-slave-03cd krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:21.596900 3512 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:41:21.598003 3512 webserver.cc:460] Webserver started at http://127.0.98.1:37409/ using document root <none> and password file <none>
I20260619 16:41:21.598712 3512 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:21.598806 3512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:21.599052 3512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:21.601263 3512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/data/instance:
uuid: "8be25d7ed05d465199f2742863255aeb"
format_stamp: "Formatted at 2026-06-19 16:41:21 on dist-test-slave-03cd"
server_key: "02e70ce314f0a470f25d8d3cc74fdf33"
server_key_iv: "fd379c79946bd11439065a13d7632aa0"
server_key_version: "encryptionkey@0"
I20260619 16:41:21.601867 3512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8be25d7ed05d465199f2742863255aeb"
format_stamp: "Formatted at 2026-06-19 16:41:21 on dist-test-slave-03cd"
server_key: "02e70ce314f0a470f25d8d3cc74fdf33"
server_key_iv: "fd379c79946bd11439065a13d7632aa0"
server_key_version: "encryptionkey@0"
I20260619 16:41:21.605940 3512 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:41:21.608805 3527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:21.610276 3512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:41:21.610460 3512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "8be25d7ed05d465199f2742863255aeb"
format_stamp: "Formatted at 2026-06-19 16:41:21 on dist-test-slave-03cd"
server_key: "02e70ce314f0a470f25d8d3cc74fdf33"
server_key_iv: "fd379c79946bd11439065a13d7632aa0"
server_key_version: "encryptionkey@0"
I20260619 16:41:21.610600 3512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:21.629551 3512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:21.633524 3512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:21.633817 3512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:21.634505 3512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:21.635480 3512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:21.635558 3512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:21.635638 3512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:21.635706 3512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 16:41:21.647235 3512 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37477
I20260619 16:41:21.647320 3640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37477 every 8 connection(s)
I20260619 16:41:21.649000 3512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:41:21.649644 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 3512
I20260619 16:41:21.649767 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:41:21.650170 392 external_mini_cluster.cc:1428] Setting key 28cd26c93eda8e5ad877a716ed65f519
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Jun 19 16:41:21 dist-test-slave-03cd krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:21.670526 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:21.651820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48187 (local address 127.0.98.62:43225)
0619 16:41:21.651953 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:21.651958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:21.653126 (+ 1168us) server_negotiation.cc:407] Connection header received
0619 16:41:21.655200 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:21.655204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:21.655260 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:21.655349 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:21.657180 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:21.657746 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:21.659803 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:21.660093 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:21.660199 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:21.665750 (+ 5551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:21.665758 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:21.665764 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:21.665807 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:21.667601 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:21.668525 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:21.668528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:21.668530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:21.668613 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:21.669123 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:21.669127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:21.669128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:21.669363 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:21.669447 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:21.670262 (+ 815us) server_negotiation.cc:299] Negotiation successful
0619 16:41:21.670394 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:41:21.672098 3643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:21.652390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43225 (local address 127.0.98.1:48187)
0619 16:41:21.652931 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:21.652987 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:21.654926 (+ 1939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:21.655566 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:21.655579 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:21.656200 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:21.656976 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:21.657001 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:21.657898 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:21.657903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:21.659510 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:21.659525 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:21.660296 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:21.660300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:21.660555 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:21.661249 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:21.661284 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:21.665486 (+ 4202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:21.667881 (+ 2395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:21.667892 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:21.667915 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:21.668363 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:21.668728 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:21.668735 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:21.668739 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:21.668970 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:21.669481 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:21.669493 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:21.669914 (+ 421us) client_negotiation.cc:769] Sending connection context
0619 16:41:21.670185 (+ 271us) client_negotiation.cc:240] Negotiation successful
0619 16:41:21.670649 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":170,"threads_started":1}
I20260619 16:41:21.673815 3641 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43225
I20260619 16:41:21.674173 3641 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:21.674793 3641 heartbeater.cc:511] Master 127.0.98.62:43225 requested a full tablet report, sending...
I20260619 16:41:21.676810 3450 ts_manager.cc:194] Registered new tserver with Master: 8be25d7ed05d465199f2742863255aeb (127.0.98.1:37477)
I20260619 16:41:21.678256 3450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:41:21.685551 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43225
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:21.810926 3646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:21.811249 3646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:21.811296 3646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:21.816094 3646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:21.816184 3646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:21.816250 3646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:41:21.822451 3646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:21.823717 3646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:21.824818 3646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:21.835767 3654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:21.835784 3646 server_base.cc:1034] running on GCE node
W20260619 16:41:21.835831 3652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:21.836120 3651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:21.836586 3646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:21.837329 3646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:21.838541 3646 hybrid_clock.cc:648] HybridClock initialized: now 1781887281838524 us; error 37 us; skew 500 ppm
Jun 19 16:41:21 dist-test-slave-03cd krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:21.841746 3646 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:41:21.842856 3646 webserver.cc:460] Webserver started at http://127.0.98.2:37089/ using document root <none> and password file <none>
I20260619 16:41:21.843479 3646 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:21.843577 3646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:21.843851 3646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:21.846454 3646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/data/instance:
uuid: "0a1868a5976f430690695e7124d5046e"
format_stamp: "Formatted at 2026-06-19 16:41:21 on dist-test-slave-03cd"
server_key: "939f92f154c47a15676e8173acd9e3a9"
server_key_iv: "1c03aa879f77c1d45a7f8fab36e1aa62"
server_key_version: "encryptionkey@0"
I20260619 16:41:21.847146 3646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0a1868a5976f430690695e7124d5046e"
format_stamp: "Formatted at 2026-06-19 16:41:21 on dist-test-slave-03cd"
server_key: "939f92f154c47a15676e8173acd9e3a9"
server_key_iv: "1c03aa879f77c1d45a7f8fab36e1aa62"
server_key_version: "encryptionkey@0"
I20260619 16:41:21.851437 3646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 16:41:21.854386 3661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:21.855973 3646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:41:21.856101 3646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "0a1868a5976f430690695e7124d5046e"
format_stamp: "Formatted at 2026-06-19 16:41:21 on dist-test-slave-03cd"
server_key: "939f92f154c47a15676e8173acd9e3a9"
server_key_iv: "1c03aa879f77c1d45a7f8fab36e1aa62"
server_key_version: "encryptionkey@0"
I20260619 16:41:21.856204 3646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:21.866569 3646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:21.870024 3646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:21.870194 3646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:21.870776 3646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:21.871665 3646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:21.871764 3646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:21.871812 3646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:21.871829 3646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:21.882555 3646 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35741
I20260619 16:41:21.882577 3774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35741 every 8 connection(s)
I20260619 16:41:21.883587 3646 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:41:21.893226 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 3646
I20260619 16:41:21.893442 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:41:21.893783 392 external_mini_cluster.cc:1428] Setting key b9b5b8db7eee503f4d44ab5986f3c983
Jun 19 16:41:21 dist-test-slave-03cd krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:21.901132 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:21.885964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36361 (local address 127.0.98.62:43225)
0619 16:41:21.886131 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:21.886135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:21.887188 (+ 1053us) server_negotiation.cc:407] Connection header received
0619 16:41:21.888865 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:21.888869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:21.888940 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:21.889063 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:21.890394 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:21.890997 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:21.892649 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:21.892972 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:21.893113 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:21.896224 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:21.896228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:21.896230 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:21.896254 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:21.898383 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:21.899117 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:21.899121 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:21.899124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:21.899219 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:21.899766 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:21.899770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:21.899771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:21.900006 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:21.900089 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:21.900869 (+ 780us) server_negotiation.cc:299] Negotiation successful
0619 16:41:21.901003 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 16:41:21.902745 3777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:21.886373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43225 (local address 127.0.98.2:36361)
0619 16:41:21.886956 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:21.887010 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:21.888595 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:21.889260 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:21.889269 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:21.889694 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:21.890223 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:21.890238 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:21.891156 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:21.891162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:21.892325 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:21.892336 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:21.893167 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:21.893170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:21.893340 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:21.894140 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:21.894170 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:21.896017 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:21.898558 (+ 2541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:21.898565 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:21.898581 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:21.898897 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:21.899350 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:21.899356 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:21.899361 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:21.899610 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:21.900123 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:21.900134 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:21.900554 (+ 420us) client_negotiation.cc:769] Sending connection context
0619 16:41:21.900801 (+ 247us) client_negotiation.cc:240] Negotiation successful
0619 16:41:21.901351 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":143,"threads_started":1}
I20260619 16:41:21.904376 3775 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43225
I20260619 16:41:21.904687 3775 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:21.905309 3775 heartbeater.cc:511] Master 127.0.98.62:43225 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:41:21.906795 3450 ts_manager.cc:194] Registered new tserver with Master: 0a1868a5976f430690695e7124d5046e (127.0.98.2:35741)
I20260619 16:41:21.907430 3450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36361
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:41:21.928679 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43225
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:22.055219 3780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:22.055541 3780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:22.055589 3780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:22.060055 3780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:22.060153 3780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:22.060223 3780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:41:22.066037 3780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:22.067224 3780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:22.068362 3780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:22.080274 3786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:22.080308 3788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:22.080433 3780 server_base.cc:1034] running on GCE node
W20260619 16:41:22.080258 3785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:22.081061 3780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:22.081892 3780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:22.083120 3780 hybrid_clock.cc:648] HybridClock initialized: now 1781887282083097 us; error 35 us; skew 500 ppm
Jun 19 16:41:22 dist-test-slave-03cd krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:22.086611 3780 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:41:22.087769 3780 webserver.cc:460] Webserver started at http://127.0.98.3:35357/ using document root <none> and password file <none>
I20260619 16:41:22.088455 3780 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:22.088533 3780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:22.088758 3780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:22.091348 3780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/data/instance:
uuid: "0f14bafcc76549488c09f086ad87dd58"
format_stamp: "Formatted at 2026-06-19 16:41:22 on dist-test-slave-03cd"
server_key: "ad3758a53813e17eedabf891a46766d5"
server_key_iv: "efdf936d627f267e177e15bbe962862f"
server_key_version: "encryptionkey@0"
I20260619 16:41:22.092089 3780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0f14bafcc76549488c09f086ad87dd58"
format_stamp: "Formatted at 2026-06-19 16:41:22 on dist-test-slave-03cd"
server_key: "ad3758a53813e17eedabf891a46766d5"
server_key_iv: "efdf936d627f267e177e15bbe962862f"
server_key_version: "encryptionkey@0"
I20260619 16:41:22.096236 3780 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:41:22.099191 3795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:22.100806 3780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:41:22.100935 3780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "0f14bafcc76549488c09f086ad87dd58"
format_stamp: "Formatted at 2026-06-19 16:41:22 on dist-test-slave-03cd"
server_key: "ad3758a53813e17eedabf891a46766d5"
server_key_iv: "efdf936d627f267e177e15bbe962862f"
server_key_version: "encryptionkey@0"
I20260619 16:41:22.101115 3780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:22.116704 3780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:22.120606 3780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:22.120875 3780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:22.121548 3780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:22.122543 3780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:22.122594 3780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:22.122637 3780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:22.122699 3780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:22.134495 3780 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44819
I20260619 16:41:22.134539 3908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44819 every 8 connection(s)
I20260619 16:41:22.135538 3780 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:41:22.137461 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 3780
I20260619 16:41:22.137598 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887259574980-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:41:22.137938 392 external_mini_cluster.cc:1428] Setting key 871d728f1239cb54c781d2bb8e4d4cff
Jun 19 16:41:22 dist-test-slave-03cd krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:22.153968 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:22.137664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33543 (local address 127.0.98.62:43225)
0619 16:41:22.137836 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:22.137840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:22.139273 (+ 1433us) server_negotiation.cc:407] Connection header received
0619 16:41:22.141262 (+ 1989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:22.141265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:22.141330 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:22.141406 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:22.142892 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:22.143474 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:22.144799 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:22.145057 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:22.145283 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:22.149160 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:22.149171 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:22.149175 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:22.149210 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:22.150875 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:22.151623 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:22.151626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:22.151628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:22.151906 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:22.152471 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:22.152474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:22.152477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:22.152788 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:22.152899 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:22.153659 (+ 760us) server_negotiation.cc:299] Negotiation successful
0619 16:41:22.153846 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:41:22.154868 3911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:22.138197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43225 (local address 127.0.98.3:33543)
0619 16:41:22.139086 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:22.139169 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:22.140876 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:22.141687 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:22.141696 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:22.142159 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:22.142694 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:22.142708 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:22.143613 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:22.143615 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:22.144649 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:22.144661 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:22.145367 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:22.145370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:22.145544 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:22.146769 (+ 1225us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:22.146788 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:22.148697 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:22.151057 (+ 2360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:22.151063 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:22.151092 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:22.151469 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:22.152077 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:22.152080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:22.152082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:22.152328 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:22.152928 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:22.152936 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:22.153318 (+ 382us) client_negotiation.cc:769] Sending connection context
0619 16:41:22.153532 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 16:41:22.153933 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":181,"threads_started":1}
I20260619 16:41:22.156315 3909 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43225
I20260619 16:41:22.156596 3909 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:22.157188 3909 heartbeater.cc:511] Master 127.0.98.62:43225 requested a full tablet report, sending...
I20260619 16:41:22.158464 3450 ts_manager.cc:194] Registered new tserver with Master: 0f14bafcc76549488c09f086ad87dd58 (127.0.98.3:44819)
I20260619 16:41:22.159138 3450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33543
I20260619 16:41:22.166730 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:41:22 dist-test-slave-03cd krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887282, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 16:41:22 dist-test-slave-03cd krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887282, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:22.197032 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:22.187581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59612 (local address 127.0.98.62:43225)
0619 16:41:22.187764 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:22.187771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:22.188008 (+ 237us) server_negotiation.cc:407] Connection header received
0619 16:41:22.188182 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:22.188190 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:22.188235 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:22.188330 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:22.189244 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:22.189776 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:22.190684 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:22.190910 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:22.191022 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:22.193656 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:22.193660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:22.193662 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:22.193692 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:22.195178 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:22.195654 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:22.195692 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:22.195695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:22.195785 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:22.196092 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:22.196096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:22.196098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:22.196343 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:22.196488 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:22.196727 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 16:41:22.196845 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 16:41:22.197443 3450 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59612
I20260619 16:41:22.198319 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 3512
I20260619 16:41:22.204066 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 3646
I20260619 16:41:22.210080 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 3780
I20260619 16:41:22.215528 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 3418
2026-06-19T16:41:22Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2353 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:41:22 dist-test-slave-03cd krb5kdc[3921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:41:22 dist-test-slave-03cd krb5kdc[3921](info): set up 2 sockets
Jun 19 16:41:22 dist-test-slave-03cd krb5kdc[3921](info): commencing operation
krb5kdc: starting...
W20260619 16:41:23.403143 392 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260619 16:41:23.717703 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.314s user 0.000s sys 0.003s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260619 16:41:23.882277 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.164s user 0.000s sys 0.003s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260619 16:41:24.184428 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.302s user 0.000s sys 0.003s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/test-user.keytab.
W20260619 16:41:24.604229 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.420s user 0.000s sys 0.003s
Jun 19 16:41:24 dist-test-slave-03cd krb5kdc[3921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:41:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:41:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:41:24.661105 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35877
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:24.802042 3934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:24.802496 3934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:24.802584 3934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:24.807377 3934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:41:24.807538 3934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:24.807578 3934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:24.807612 3934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:41:24.807649 3934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:41:24.814466 3934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:24.816167 3934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:24.817576 3934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:24.828441 3942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:24.828441 3939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:24.828486 3940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:24.828941 3934 server_base.cc:1034] running on GCE node
I20260619 16:41:24.829661 3934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:24.831139 3934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:24.832396 3934 hybrid_clock.cc:648] HybridClock initialized: now 1781887284832362 us; error 102 us; skew 500 ppm
Jun 19 16:41:24 dist-test-slave-03cd krb5kdc[3921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:24.836180 3934 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:41:24.837414 3934 webserver.cc:460] Webserver started at http://127.0.98.62:41867/ using document root <none> and password file <none>
I20260619 16:41:24.838100 3934 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:24.838218 3934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:24.838483 3934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:24.841289 3934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "eea7539fef66435c9fb4aa53b45d80fc"
format_stamp: "Formatted at 2026-06-19 16:41:24 on dist-test-slave-03cd"
server_key: "853181cac5ef5cc56ca54f4e543275d9"
server_key_iv: "69c0942dd475909878304379af925859"
server_key_version: "encryptionkey@0"
I20260619 16:41:24.842006 3934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "eea7539fef66435c9fb4aa53b45d80fc"
format_stamp: "Formatted at 2026-06-19 16:41:24 on dist-test-slave-03cd"
server_key: "853181cac5ef5cc56ca54f4e543275d9"
server_key_iv: "69c0942dd475909878304379af925859"
server_key_version: "encryptionkey@0"
I20260619 16:41:24.847348 3934 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 16:41:24.851048 3949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:24.853166 3934 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:41:24.853396 3934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "eea7539fef66435c9fb4aa53b45d80fc"
format_stamp: "Formatted at 2026-06-19 16:41:24 on dist-test-slave-03cd"
server_key: "853181cac5ef5cc56ca54f4e543275d9"
server_key_iv: "69c0942dd475909878304379af925859"
server_key_version: "encryptionkey@0"
I20260619 16:41:24.853540 3934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:24.878563 3934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:24.883040 3934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:24.883275 3934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:24.894733 3934 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35877
I20260619 16:41:24.894754 4001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35877 every 8 connection(s)
I20260619 16:41:24.896148 3934 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:41:24.899652 4002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:24.900421 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 3934
I20260619 16:41:24.900555 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:41:24.900882 392 external_mini_cluster.cc:1428] Setting key af1babe0efc576ef468f65647e185ff3
I20260619 16:41:24.907649 4002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eea7539fef66435c9fb4aa53b45d80fc: Bootstrap starting.
I20260619 16:41:24.911398 4002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eea7539fef66435c9fb4aa53b45d80fc: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:24.912531 4002 log.cc:824] T 00000000000000000000000000000000 P eea7539fef66435c9fb4aa53b45d80fc: Log is configured to *not* fsync() on all Append() calls
Jun 19 16:41:24 dist-test-slave-03cd krb5kdc[3921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:24.915136 4002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eea7539fef66435c9fb4aa53b45d80fc: No bootstrap required, opened a new log
I20260619 16:41:24.919764 4002 raft_consensus.cc:348] T 00000000000000000000000000000000 P eea7539fef66435c9fb4aa53b45d80fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea7539fef66435c9fb4aa53b45d80fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35877 } }
I20260619 16:41:24.920074 4002 raft_consensus.cc:374] T 00000000000000000000000000000000 P eea7539fef66435c9fb4aa53b45d80fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:24.920113 4002 raft_consensus.cc:729] T 00000000000000000000000000000000 P eea7539fef66435c9fb4aa53b45d80fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea7539fef66435c9fb4aa53b45d80fc, State: Initialized, Role: FOLLOWER
I20260619 16:41:24.920389 4005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:24.902850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42994 (local address 127.0.98.62:35877)
0619 16:41:24.903519 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:24.903547 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:24.903600 (+ 53us) server_negotiation.cc:407] Connection header received
0619 16:41:24.904550 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:24.904574 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:24.905071 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:24.905547 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:24.906617 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:24.907836 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:24.909428 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:24.910076 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:24.910272 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:24.913816 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:24.913824 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:24.913847 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:24.913884 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:24.916147 (+ 2263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:24.916777 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:24.916783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:24.916789 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:24.916969 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:24.917372 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:24.917375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:24.917377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:24.917994 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:24.918178 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:24.918608 (+ 430us) server_negotiation.cc:299] Negotiation successful
0619 16:41:24.919090 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":181,"threads_started":1}
I20260619 16:41:24.920734 4002 consensus_queue.cc:260] T 00000000000000000000000000000000 P eea7539fef66435c9fb4aa53b45d80fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea7539fef66435c9fb4aa53b45d80fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35877 } }
I20260619 16:41:24.920907 4002 raft_consensus.cc:388] T 00000000000000000000000000000000 P eea7539fef66435c9fb4aa53b45d80fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:41:24.920984 4002 raft_consensus.cc:482] T 00000000000000000000000000000000 P eea7539fef66435c9fb4aa53b45d80fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:41:24.921093 4002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eea7539fef66435c9fb4aa53b45d80fc [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:24.922272 4002 raft_consensus.cc:504] T 00000000000000000000000000000000 P eea7539fef66435c9fb4aa53b45d80fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea7539fef66435c9fb4aa53b45d80fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35877 } }
I20260619 16:41:24.922706 4002 leader_election.cc:302] T 00000000000000000000000000000000 P eea7539fef66435c9fb4aa53b45d80fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eea7539fef66435c9fb4aa53b45d80fc; no voters:
I20260619 16:41:24.923092 4002 leader_election.cc:288] T 00000000000000000000000000000000 P eea7539fef66435c9fb4aa53b45d80fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:41:24.923267 4007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eea7539fef66435c9fb4aa53b45d80fc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:24.923595 4007 raft_consensus.cc:686] T 00000000000000000000000000000000 P eea7539fef66435c9fb4aa53b45d80fc [term 1 LEADER]: Becoming Leader. State: Replica: eea7539fef66435c9fb4aa53b45d80fc, State: Running, Role: LEADER
I20260619 16:41:24.924063 4007 consensus_queue.cc:237] T 00000000000000000000000000000000 P eea7539fef66435c9fb4aa53b45d80fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea7539fef66435c9fb4aa53b45d80fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35877 } }
I20260619 16:41:24.924722 4002 sys_catalog.cc:564] T 00000000000000000000000000000000 P eea7539fef66435c9fb4aa53b45d80fc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:41:24.926172 4008 sys_catalog.cc:455] T 00000000000000000000000000000000 P eea7539fef66435c9fb4aa53b45d80fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eea7539fef66435c9fb4aa53b45d80fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea7539fef66435c9fb4aa53b45d80fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35877 } } }
I20260619 16:41:24.926390 4008 sys_catalog.cc:458] T 00000000000000000000000000000000 P eea7539fef66435c9fb4aa53b45d80fc [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:24.927048 4016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:41:24.927119 4009 sys_catalog.cc:455] T 00000000000000000000000000000000 P eea7539fef66435c9fb4aa53b45d80fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader eea7539fef66435c9fb4aa53b45d80fc. Latest consensus state: current_term: 1 leader_uuid: "eea7539fef66435c9fb4aa53b45d80fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea7539fef66435c9fb4aa53b45d80fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35877 } } }
I20260619 16:41:24.928036 4009 sys_catalog.cc:458] T 00000000000000000000000000000000 P eea7539fef66435c9fb4aa53b45d80fc [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:24.931124 4016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:41:24.940193 4016 catalog_manager.cc:1347] Generated new cluster ID: 6eefd8632a274ae3839c289ebb56c454
I20260619 16:41:24.940330 4016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:41:24.953692 4016 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:41:24.955128 4016 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:41:24.965180 4016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eea7539fef66435c9fb4aa53b45d80fc: Generated new TSK 0
I20260619 16:41:24.966404 4016 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:41:25.020375 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35877
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:25.175921 4028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:25.176399 4028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:25.176481 4028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:25.181694 4028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:25.181834 4028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:25.181954 4028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:41:25.188511 4028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:25.190184 4028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:25.191895 4028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:25.205147 4034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:25.205178 4033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:25.205147 4036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:25.205658 4028 server_base.cc:1034] running on GCE node
I20260619 16:41:25.206285 4028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:25.207118 4028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:25.208441 4028 hybrid_clock.cc:648] HybridClock initialized: now 1781887285208390 us; error 89 us; skew 500 ppm
Jun 19 16:41:25 dist-test-slave-03cd krb5kdc[3921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:25.212466 4028 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:41:25.213701 4028 webserver.cc:460] Webserver started at http://127.0.98.1:41811/ using document root <none> and password file <none>
I20260619 16:41:25.214424 4028 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:25.214535 4028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:25.214811 4028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:25.217300 4028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-0/data/instance:
uuid: "7556dba3983449108d69d0813a521b60"
format_stamp: "Formatted at 2026-06-19 16:41:25 on dist-test-slave-03cd"
server_key: "b893f17fa216340b6baa8afcd3984852"
server_key_iv: "cdfff7a5e7ea0d68f63ea04d992b037d"
server_key_version: "encryptionkey@0"
I20260619 16:41:25.217962 4028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7556dba3983449108d69d0813a521b60"
format_stamp: "Formatted at 2026-06-19 16:41:25 on dist-test-slave-03cd"
server_key: "b893f17fa216340b6baa8afcd3984852"
server_key_iv: "cdfff7a5e7ea0d68f63ea04d992b037d"
server_key_version: "encryptionkey@0"
I20260619 16:41:25.222765 4028 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 16:41:25.226554 4043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:25.228581 4028 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 16:41:25.228776 4028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "7556dba3983449108d69d0813a521b60"
format_stamp: "Formatted at 2026-06-19 16:41:25 on dist-test-slave-03cd"
server_key: "b893f17fa216340b6baa8afcd3984852"
server_key_iv: "cdfff7a5e7ea0d68f63ea04d992b037d"
server_key_version: "encryptionkey@0"
I20260619 16:41:25.228924 4028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:25.254707 4028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:25.259030 4028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:25.259284 4028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:25.260486 4028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:25.261937 4028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:25.262022 4028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:25.262085 4028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:25.262126 4028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:25.276666 4028 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40943
I20260619 16:41:25.276690 4156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40943 every 8 connection(s)
I20260619 16:41:25.277912 4028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:41:25.281502 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 4028
I20260619 16:41:25.281975 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:41:25.282584 392 external_mini_cluster.cc:1428] Setting key 92b9db55883c1e214180a0d6f9b26278
Jun 19 16:41:25 dist-test-slave-03cd krb5kdc[3921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:41:25.297259 4005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:25.281025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57409 (local address 127.0.98.62:35877)
0619 16:41:25.281308 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:25.281326 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:25.282181 (+ 855us) server_negotiation.cc:407] Connection header received
0619 16:41:25.284362 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:25.284421 (+ 59us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:25.284509 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:25.284650 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:25.286732 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:25.287423 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:25.288806 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:25.289064 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:25.289195 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:25.292723 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:25.292727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:25.292729 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:25.292759 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:25.294735 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:25.295366 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:25.295370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:25.295372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:25.295460 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:25.295929 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:25.295932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:25.295934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:25.296152 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:25.296271 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:25.296904 (+ 633us) server_negotiation.cc:299] Negotiation successful
0619 16:41:25.297068 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260619 16:41:25.298403 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:25.281451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35877 (local address 127.0.98.1:57409)
0619 16:41:25.281995 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:25.282054 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:25.284060 (+ 2006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:25.285000 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:25.285011 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:25.285654 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:25.286494 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:25.286520 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:25.287571 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:25.287573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:25.288631 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:25.288644 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:25.289250 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:25.289254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:25.289455 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:25.290425 (+ 970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:25.290458 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:25.292489 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:25.294886 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:25.294894 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:25.294916 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:25.295238 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:25.295564 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:25.295567 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:25.295569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:25.295819 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:25.296270 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:25.296280 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:25.296585 (+ 305us) client_negotiation.cc:769] Sending connection context
0619 16:41:25.296803 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 16:41:25.297213 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":152,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 16:41:25.300627 4157 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35877
I20260619 16:41:25.301107 4157 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:25.302445 4157 heartbeater.cc:511] Master 127.0.98.62:35877 requested a full tablet report, sending...
I20260619 16:41:25.305282 3966 ts_manager.cc:194] Registered new tserver with Master: 7556dba3983449108d69d0813a521b60 (127.0.98.1:40943)
I20260619 16:41:25.307610 3966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57409
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:41:25.323625 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35877
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:25.481142 4162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:25.481582 4162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:25.481660 4162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:25.486928 4162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:25.487087 4162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:25.487206 4162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:41:25.494009 4162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:25.495596 4162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:25.497043 4162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:25.509672 4170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:25.509678 4167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:25.509672 4168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:25.510298 4162 server_base.cc:1034] running on GCE node
I20260619 16:41:25.510823 4162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:25.511590 4162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:25.512806 4162 hybrid_clock.cc:648] HybridClock initialized: now 1781887285512788 us; error 54 us; skew 500 ppm
Jun 19 16:41:25 dist-test-slave-03cd krb5kdc[3921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:25.515962 4162 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:41:25.516964 4162 webserver.cc:460] Webserver started at http://127.0.98.2:46441/ using document root <none> and password file <none>
I20260619 16:41:25.517580 4162 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:25.517644 4162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:25.517827 4162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:25.520112 4162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-1/data/instance:
uuid: "4cddf45eca6841a1869a51a5502f77d6"
format_stamp: "Formatted at 2026-06-19 16:41:25 on dist-test-slave-03cd"
server_key: "c14eab60e39afb86485f2cf4dfcb87e3"
server_key_iv: "563c6e68365f6a3fd354a7403d1fc7f7"
server_key_version: "encryptionkey@0"
I20260619 16:41:25.520690 4162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4cddf45eca6841a1869a51a5502f77d6"
format_stamp: "Formatted at 2026-06-19 16:41:25 on dist-test-slave-03cd"
server_key: "c14eab60e39afb86485f2cf4dfcb87e3"
server_key_iv: "563c6e68365f6a3fd354a7403d1fc7f7"
server_key_version: "encryptionkey@0"
I20260619 16:41:25.525323 4162 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:41:25.528875 4177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:25.530627 4162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:41:25.530814 4162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "4cddf45eca6841a1869a51a5502f77d6"
format_stamp: "Formatted at 2026-06-19 16:41:25 on dist-test-slave-03cd"
server_key: "c14eab60e39afb86485f2cf4dfcb87e3"
server_key_iv: "563c6e68365f6a3fd354a7403d1fc7f7"
server_key_version: "encryptionkey@0"
I20260619 16:41:25.530973 4162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:25.549561 4162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:25.554497 4162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:25.554692 4162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:25.556038 4162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:25.557163 4162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:25.557219 4162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:25.557269 4162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:25.557338 4162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:25.568615 4162 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35235
I20260619 16:41:25.568670 4290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35235 every 8 connection(s)
I20260619 16:41:25.569903 4162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:41:25.574196 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 4162
I20260619 16:41:25.574359 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:41:25.574772 392 external_mini_cluster.cc:1428] Setting key eb64814ac9b0d1ac627506def5e1adc9
Jun 19 16:41:25 dist-test-slave-03cd krb5kdc[3921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 16:41:25.589903 4005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:25.572456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34035 (local address 127.0.98.62:35877)
0619 16:41:25.572660 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:25.572666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:25.573610 (+ 944us) server_negotiation.cc:407] Connection header received
0619 16:41:25.575637 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:25.575642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:25.575747 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:25.575857 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:25.577858 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:25.578465 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:25.580107 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:25.580381 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:25.580514 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:25.584341 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:25.584345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:25.584349 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:25.584391 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:25.586460 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:25.587327 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:25.587330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:25.587332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:25.587453 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:25.588072 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:25.588108 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:25.588113 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:25.588378 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:25.588480 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:25.589459 (+ 979us) server_negotiation.cc:299] Negotiation successful
0619 16:41:25.589667 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:41:25.590762 4293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:25.572813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35877 (local address 127.0.98.2:34035)
0619 16:41:25.573380 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:25.573439 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:25.575360 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:25.576151 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:25.576163 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:25.576750 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:25.577632 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:25.577662 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:25.578633 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:25.578640 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:25.579936 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:25.579949 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:25.580575 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:25.580580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:25.580801 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:25.581688 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:25.581720 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:25.583993 (+ 2273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:25.586679 (+ 2686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:25.586686 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:25.586703 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:25.587175 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:25.587595 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:25.587598 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:25.587599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:25.587854 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:25.588504 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:25.588510 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:25.589011 (+ 501us) client_negotiation.cc:769] Sending connection context
0619 16:41:25.589308 (+ 297us) client_negotiation.cc:240] Negotiation successful
0619 16:41:25.589680 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":118,"threads_started":1}
I20260619 16:41:25.592406 4291 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35877
I20260619 16:41:25.592746 4291 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:25.593451 4291 heartbeater.cc:511] Master 127.0.98.62:35877 requested a full tablet report, sending...
I20260619 16:41:25.595084 3966 ts_manager.cc:194] Registered new tserver with Master: 4cddf45eca6841a1869a51a5502f77d6 (127.0.98.2:35235)
I20260619 16:41:25.595928 3966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:41:25.612363 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35877
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:25.748917 4296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:25.749342 4296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:25.749440 4296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:25.754115 4296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:25.754218 4296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:25.754292 4296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:41:25.762003 4296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:25.763451 4296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:25.764881 4296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:25.777128 4304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:25.777165 4301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:25.777163 4302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:25.777892 4296 server_base.cc:1034] running on GCE node
I20260619 16:41:25.778398 4296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:25.779104 4296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:25.780329 4296 hybrid_clock.cc:648] HybridClock initialized: now 1781887285780314 us; error 49 us; skew 500 ppm
Jun 19 16:41:25 dist-test-slave-03cd krb5kdc[3921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:25.783586 4296 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:41:25.784669 4296 webserver.cc:460] Webserver started at http://127.0.98.3:35531/ using document root <none> and password file <none>
I20260619 16:41:25.785266 4296 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:25.785333 4296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:25.785521 4296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:25.787839 4296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-2/data/instance:
uuid: "d4aeb4e414ed4777a0c8574d377a306a"
format_stamp: "Formatted at 2026-06-19 16:41:25 on dist-test-slave-03cd"
server_key: "4f6df67ddec05715fe276b61dee53241"
server_key_iv: "d4de14663f79bd909b38ada9367a876b"
server_key_version: "encryptionkey@0"
I20260619 16:41:25.788427 4296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d4aeb4e414ed4777a0c8574d377a306a"
format_stamp: "Formatted at 2026-06-19 16:41:25 on dist-test-slave-03cd"
server_key: "4f6df67ddec05715fe276b61dee53241"
server_key_iv: "d4de14663f79bd909b38ada9367a876b"
server_key_version: "encryptionkey@0"
I20260619 16:41:25.793083 4296 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:41:25.796363 4311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:25.798137 4296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:41:25.798333 4296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "d4aeb4e414ed4777a0c8574d377a306a"
format_stamp: "Formatted at 2026-06-19 16:41:25 on dist-test-slave-03cd"
server_key: "4f6df67ddec05715fe276b61dee53241"
server_key_iv: "d4de14663f79bd909b38ada9367a876b"
server_key_version: "encryptionkey@0"
I20260619 16:41:25.798509 4296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:25.829087 4296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:25.832531 4296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:25.832744 4296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:25.833462 4296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:25.834422 4296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:25.834491 4296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:25.834565 4296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:25.834609 4296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:25.845480 4296 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34445
I20260619 16:41:25.845590 4424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34445 every 8 connection(s)
I20260619 16:41:25.846563 4296 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:41:25.851491 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 4296
I20260619 16:41:25.851639 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887259574980-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:41:25.852041 392 external_mini_cluster.cc:1428] Setting key 6547dc57f4ea7d3fd40d414bf4cf186b
Jun 19 16:41:25 dist-test-slave-03cd krb5kdc[3921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:25.866888 4005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:25.849323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39941 (local address 127.0.98.62:35877)
0619 16:41:25.849534 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:25.849538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:25.850570 (+ 1032us) server_negotiation.cc:407] Connection header received
0619 16:41:25.852868 (+ 2298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:25.852871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:25.852934 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:25.853045 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:25.854764 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:25.855431 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:25.856956 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:25.857237 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:25.857364 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:25.860993 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:25.860998 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:25.861001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:25.861045 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:25.863228 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:25.864485 (+ 1257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:25.864489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:25.864490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:25.864611 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:25.865073 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:25.865079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:25.865084 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:25.865364 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:25.865511 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:25.866468 (+ 957us) server_negotiation.cc:299] Negotiation successful
0619 16:41:25.866630 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 16:41:25.868180 4427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:25.849703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35877 (local address 127.0.98.3:39941)
0619 16:41:25.850326 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:25.850395 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:25.852414 (+ 2019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:25.853295 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:25.853304 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:25.853810 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:25.854581 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:25.854602 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:25.855585 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:25.855590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:25.856790 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:25.856801 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:25.857426 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:25.857430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:25.857712 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:25.858564 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:25.858586 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:25.860733 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:25.863619 (+ 2886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:25.863630 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:25.863699 (+ 69us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:25.864308 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:25.864753 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:25.864756 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:25.864758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:25.864929 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:25.865527 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:25.865541 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:25.865995 (+ 454us) client_negotiation.cc:769] Sending connection context
0619 16:41:25.866420 (+ 425us) client_negotiation.cc:240] Negotiation successful
0619 16:41:25.866911 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":122,"threads_started":1}
I20260619 16:41:25.870319 4425 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35877
I20260619 16:41:25.870692 4425 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:25.871454 4425 heartbeater.cc:511] Master 127.0.98.62:35877 requested a full tablet report, sending...
I20260619 16:41:25.873209 3966 ts_manager.cc:194] Registered new tserver with Master: d4aeb4e414ed4777a0c8574d377a306a (127.0.98.3:34445)
I20260619 16:41:25.873946 3966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39941
I20260619 16:41:25.879907 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:41:25 dist-test-slave-03cd krb5kdc[3921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:41:25 dist-test-slave-03cd krb5kdc[3921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:41:25.914072 4432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:25.901106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59326 (local address 127.0.98.1:40943)
0619 16:41:25.901518 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:25.901525 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:25.901561 (+ 36us) server_negotiation.cc:407] Connection header received
0619 16:41:25.901731 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:25.901737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:25.901958 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:25.902153 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:25.903435 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:25.904469 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:25.905929 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:25.906345 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:25.906466 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:25.910074 (+ 3608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:25.910088 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:25.910103 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:25.910139 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:25.912048 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:25.912594 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:25.912600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:25.912601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:25.912686 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:25.912985 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:25.912990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:25.912991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:25.913278 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:25.913509 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:25.913723 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 16:41:25.913882 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":70,"threads_started":1}
Jun 19 16:41:25 dist-test-slave-03cd krb5kdc[3921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:25.927919 4005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:25.917811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43006 (local address 127.0.98.62:35877)
0619 16:41:25.918047 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:25.918052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:25.918186 (+ 134us) server_negotiation.cc:407] Connection header received
0619 16:41:25.918285 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:25.918291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:25.918351 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:25.918444 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:25.919080 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:25.919771 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:25.920617 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:25.920846 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:25.920952 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:25.923924 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:25.923929 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:25.923931 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:25.923966 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:25.925818 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:25.926383 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:25.926386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:25.926387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:25.926495 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:25.926777 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:25.926780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:25.926781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:25.927070 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:25.927189 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:25.927491 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 16:41:25.927619 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260619 16:41:25.928376 3966 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43006
I20260619 16:41:25.929271 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 4028
I20260619 16:41:25.936738 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 4162
I20260619 16:41:25.943292 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 4296
I20260619 16:41:25.949831 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 3934
2026-06-19T16:41:25Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3735 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:41:26 dist-test-slave-03cd krb5kdc[4437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:41:26 dist-test-slave-03cd krb5kdc[4437](info): set up 2 sockets
Jun 19 16:41:26 dist-test-slave-03cd krb5kdc[4437](info): commencing operation
krb5kdc: starting...
W20260619 16:41:27.080129 392 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:41:27 dist-test-slave-03cd krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:41:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:41:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:41:27.212561 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42321
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:27.342736 4450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:27.343061 4450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:27.343107 4450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:27.347467 4450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:41:27.347565 4450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:27.347585 4450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:27.347602 4450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:41:27.347617 4450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:41:27.353727 4450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:27.355033 4450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:27.356225 4450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:27.365796 4456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:27.365789 4455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:27.365789 4458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:27.366214 4450 server_base.cc:1034] running on GCE node
I20260619 16:41:27.366935 4450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:27.368235 4450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:27.369477 4450 hybrid_clock.cc:648] HybridClock initialized: now 1781887287369417 us; error 81 us; skew 500 ppm
Jun 19 16:41:27 dist-test-slave-03cd krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:27.376614 4450 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:41:27.377499 4450 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:27.377606 4450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:27.377903 4450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:27.380209 4450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "d7b0c7066b564a58b368550fef7e45e3"
format_stamp: "Formatted at 2026-06-19 16:41:27 on dist-test-slave-03cd"
server_key: "0495bc711c7dbaf586929bf0ad6c2265"
server_key_iv: "6f95df7ba1f45c882820932c2e8b64aa"
server_key_version: "encryptionkey@0"
I20260619 16:41:27.380833 4450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "d7b0c7066b564a58b368550fef7e45e3"
format_stamp: "Formatted at 2026-06-19 16:41:27 on dist-test-slave-03cd"
server_key: "0495bc711c7dbaf586929bf0ad6c2265"
server_key_iv: "6f95df7ba1f45c882820932c2e8b64aa"
server_key_version: "encryptionkey@0"
I20260619 16:41:27.384907 4450 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:41:27.387792 4464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:27.389242 4450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:41:27.389438 4450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "d7b0c7066b564a58b368550fef7e45e3"
format_stamp: "Formatted at 2026-06-19 16:41:27 on dist-test-slave-03cd"
server_key: "0495bc711c7dbaf586929bf0ad6c2265"
server_key_iv: "6f95df7ba1f45c882820932c2e8b64aa"
server_key_version: "encryptionkey@0"
I20260619 16:41:27.389582 4450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:27.408018 4450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:27.411897 4450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:27.412154 4450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:27.420178 4450 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42321
I20260619 16:41:27.420655 4516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42321 every 8 connection(s)
I20260619 16:41:27.421051 4450 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:41:27.424391 4517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:27.430822 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 4450
I20260619 16:41:27.430892 4517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7b0c7066b564a58b368550fef7e45e3: Bootstrap starting.
I20260619 16:41:27.431001 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:41:27.431279 392 external_mini_cluster.cc:1428] Setting key 2ebf965b365790dfacb8b1da8746084f
I20260619 16:41:27.433952 4517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7b0c7066b564a58b368550fef7e45e3: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:27.434968 4517 log.cc:824] T 00000000000000000000000000000000 P d7b0c7066b564a58b368550fef7e45e3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:27.437707 4517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7b0c7066b564a58b368550fef7e45e3: No bootstrap required, opened a new log
Jun 19 16:41:27 dist-test-slave-03cd krb5kdc[4437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:27.442397 4517 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7b0c7066b564a58b368550fef7e45e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7b0c7066b564a58b368550fef7e45e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42321 } }
I20260619 16:41:27.442773 4517 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7b0c7066b564a58b368550fef7e45e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:27.442823 4517 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7b0c7066b564a58b368550fef7e45e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b0c7066b564a58b368550fef7e45e3, State: Initialized, Role: FOLLOWER
I20260619 16:41:27.443325 4517 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7b0c7066b564a58b368550fef7e45e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7b0c7066b564a58b368550fef7e45e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42321 } }
I20260619 16:41:27.443508 4517 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7b0c7066b564a58b368550fef7e45e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:41:27.443598 4517 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7b0c7066b564a58b368550fef7e45e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:41:27.443768 4517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7b0c7066b564a58b368550fef7e45e3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:27.445245 4517 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7b0c7066b564a58b368550fef7e45e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7b0c7066b564a58b368550fef7e45e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42321 } }
I20260619 16:41:27.445705 4517 leader_election.cc:302] T 00000000000000000000000000000000 P d7b0c7066b564a58b368550fef7e45e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7b0c7066b564a58b368550fef7e45e3; no voters:
I20260619 16:41:27.446148 4517 leader_election.cc:288] T 00000000000000000000000000000000 P d7b0c7066b564a58b368550fef7e45e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:41:27.446252 4522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7b0c7066b564a58b368550fef7e45e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:27.446481 4522 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7b0c7066b564a58b368550fef7e45e3 [term 1 LEADER]: Becoming Leader. State: Replica: d7b0c7066b564a58b368550fef7e45e3, State: Running, Role: LEADER
I20260619 16:41:27.446990 4522 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7b0c7066b564a58b368550fef7e45e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7b0c7066b564a58b368550fef7e45e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42321 } }
I20260619 16:41:27.447719 4517 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7b0c7066b564a58b368550fef7e45e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:41:27.448849 4523 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7b0c7066b564a58b368550fef7e45e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7b0c7066b564a58b368550fef7e45e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7b0c7066b564a58b368550fef7e45e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42321 } } }
I20260619 16:41:27.449007 4523 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7b0c7066b564a58b368550fef7e45e3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:27.449393 4524 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7b0c7066b564a58b368550fef7e45e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7b0c7066b564a58b368550fef7e45e3. Latest consensus state: current_term: 1 leader_uuid: "d7b0c7066b564a58b368550fef7e45e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7b0c7066b564a58b368550fef7e45e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42321 } } }
I20260619 16:41:27.449496 4524 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7b0c7066b564a58b368550fef7e45e3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:27.450595 4531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:41:27.450745 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:27.432798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47702 (local address 127.0.98.62:42321)
0619 16:41:27.433296 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:27.433316 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:27.433362 (+ 46us) server_negotiation.cc:407] Connection header received
0619 16:41:27.434406 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:27.434431 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:27.434892 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:27.435310 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:27.436424 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:27.437895 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:27.438917 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:27.439719 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:27.439905 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:27.444163 (+ 4258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:27.444172 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:27.444200 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:27.444245 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:27.446565 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:27.447202 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:27.447209 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:27.447216 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:27.447344 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:27.447702 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:27.447706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:27.447710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:27.448268 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:27.449084 (+ 816us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:27.449261 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 16:41:27.449589 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":156,"threads_started":1}
I20260619 16:41:27.454469 4531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:41:27.461485 4531 catalog_manager.cc:1347] Generated new cluster ID: 7371bd3f0b674d94833c45ab066fe6c2
I20260619 16:41:27.461586 4531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:41:27.485178 4531 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:41:27.486240 4531 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:41:27.497179 4531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7b0c7066b564a58b368550fef7e45e3: Generated new TSK 0
I20260619 16:41:27.498003 4531 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:41:27.535374 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42321
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:27.661476 4543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:27.661778 4543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:27.661825 4543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:27.666090 4543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:27.666172 4543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:27.666237 4543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:41:27.671967 4543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:27.673200 4543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:27.674348 4543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:27.684662 4548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:27.684749 4551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:27.684715 4549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:27.684715 4543 server_base.cc:1034] running on GCE node
I20260619 16:41:27.685400 4543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:27.686129 4543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:27.687314 4543 hybrid_clock.cc:648] HybridClock initialized: now 1781887287687287 us; error 52 us; skew 500 ppm
Jun 19 16:41:27 dist-test-slave-03cd krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:27.690452 4543 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:41:27.691195 4543 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:27.691259 4543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:27.691462 4543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:27.693883 4543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-0/data/instance:
uuid: "d4d9806468a24073998d0ab016c5fad6"
format_stamp: "Formatted at 2026-06-19 16:41:27 on dist-test-slave-03cd"
server_key: "583eee2d120fd9f6c946c018393c3ea5"
server_key_iv: "7b612de4e09ddc41c578122e2d658cc4"
server_key_version: "encryptionkey@0"
I20260619 16:41:27.694561 4543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d4d9806468a24073998d0ab016c5fad6"
format_stamp: "Formatted at 2026-06-19 16:41:27 on dist-test-slave-03cd"
server_key: "583eee2d120fd9f6c946c018393c3ea5"
server_key_iv: "7b612de4e09ddc41c578122e2d658cc4"
server_key_version: "encryptionkey@0"
I20260619 16:41:27.698583 4543 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260619 16:41:27.702131 4557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:27.704313 4543 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:41:27.704501 4543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "d4d9806468a24073998d0ab016c5fad6"
format_stamp: "Formatted at 2026-06-19 16:41:27 on dist-test-slave-03cd"
server_key: "583eee2d120fd9f6c946c018393c3ea5"
server_key_iv: "7b612de4e09ddc41c578122e2d658cc4"
server_key_version: "encryptionkey@0"
I20260619 16:41:27.704659 4543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:27.725632 4543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:27.729339 4543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:27.729593 4543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:27.730249 4543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:27.731158 4543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:27.731235 4543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:27.731310 4543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:27.731354 4543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:27.741796 4543 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46083
I20260619 16:41:27.741813 4670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46083 every 8 connection(s)
I20260619 16:41:27.742484 4543 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:41:27.743782 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 4543
I20260619 16:41:27.743959 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:41:27.744339 392 external_mini_cluster.cc:1428] Setting key 7214c4073825f3dce36cea321316148f
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Jun 19 16:41:27 dist-test-slave-03cd krb5kdc[4437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:27.762890 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:27.744579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42827 (local address 127.0.98.62:42321)
0619 16:41:27.744743 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:27.744750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:27.745790 (+ 1040us) server_negotiation.cc:407] Connection header received
0619 16:41:27.747401 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:27.747405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:27.747468 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:27.747560 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:27.749277 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:27.749990 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:27.751274 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:27.751537 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:27.751670 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:27.758065 (+ 6395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:27.758070 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:27.758072 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:27.758102 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:27.760203 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:27.760904 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:27.760908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:27.760911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:27.761014 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:27.761573 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:27.761578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:27.761579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:27.761802 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:27.761926 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:27.762563 (+ 637us) server_negotiation.cc:299] Negotiation successful
0619 16:41:27.762716 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:41:27.763870 4673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:27.745055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42321 (local address 127.0.98.1:42827)
0619 16:41:27.745633 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:27.745693 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:27.747075 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:27.747807 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:27.747816 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:27.748307 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:27.749074 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:27.749093 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:27.750135 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:27.750139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:27.751103 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:27.751113 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:27.751747 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:27.751752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:27.751955 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:27.752676 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:27.752695 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:27.757793 (+ 5098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:27.760379 (+ 2586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:27.760387 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:27.760416 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:27.760762 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:27.761127 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:27.761131 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:27.761133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:27.761412 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:27.761943 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:27.761952 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:27.762280 (+ 328us) client_negotiation.cc:769] Sending connection context
0619 16:41:27.762508 (+ 228us) client_negotiation.cc:240] Negotiation successful
0619 16:41:27.762892 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":167,"threads_started":1}
I20260619 16:41:27.765636 4671 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42321
I20260619 16:41:27.765942 4671 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:27.766522 4671 heartbeater.cc:511] Master 127.0.98.62:42321 requested a full tablet report, sending...
I20260619 16:41:27.768674 4477 ts_manager.cc:194] Registered new tserver with Master: d4d9806468a24073998d0ab016c5fad6 (127.0.98.1:46083)
I20260619 16:41:27.770447 4477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:41:27.776232 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42321
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:27.894708 4676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:27.895000 4676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:27.895048 4676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:27.899394 4676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:27.899482 4676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:27.899549 4676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:41:27.905530 4676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:27.906728 4676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:27.907866 4676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:27.918306 4682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:27.918298 4681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:27.918444 4676 server_base.cc:1034] running on GCE node
W20260619 16:41:27.918306 4684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:27.919193 4676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:27.919965 4676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:27.921164 4676 hybrid_clock.cc:648] HybridClock initialized: now 1781887287921147 us; error 31 us; skew 500 ppm
Jun 19 16:41:27 dist-test-slave-03cd krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:27.924266 4676 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:41:27.925076 4676 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:27.925175 4676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:27.925419 4676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:27.927548 4676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-1/data/instance:
uuid: "27b1f952195c41dea8b1798cf105d0f0"
format_stamp: "Formatted at 2026-06-19 16:41:27 on dist-test-slave-03cd"
server_key: "7d52808db83e3a613f3533598d3ff134"
server_key_iv: "265580586794d41ea178a16a055c868a"
server_key_version: "encryptionkey@0"
I20260619 16:41:27.928300 4676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-1/wal/instance:
uuid: "27b1f952195c41dea8b1798cf105d0f0"
format_stamp: "Formatted at 2026-06-19 16:41:27 on dist-test-slave-03cd"
server_key: "7d52808db83e3a613f3533598d3ff134"
server_key_iv: "265580586794d41ea178a16a055c868a"
server_key_version: "encryptionkey@0"
I20260619 16:41:27.932542 4676 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:41:27.935390 4690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:27.936859 4676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:41:27.937017 4676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "27b1f952195c41dea8b1798cf105d0f0"
format_stamp: "Formatted at 2026-06-19 16:41:27 on dist-test-slave-03cd"
server_key: "7d52808db83e3a613f3533598d3ff134"
server_key_iv: "265580586794d41ea178a16a055c868a"
server_key_version: "encryptionkey@0"
I20260619 16:41:27.937170 4676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:27.952054 4676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:27.956068 4676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:27.956297 4676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:27.957073 4676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:27.958206 4676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:27.958297 4676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:27.958379 4676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:27.958431 4676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:27.968736 4676 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43639
I20260619 16:41:27.968828 4803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43639 every 8 connection(s)
I20260619 16:41:27.969467 4676 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:41:27.973805 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 4676
I20260619 16:41:27.973953 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:41:27.974224 392 external_mini_cluster.cc:1428] Setting key 5778aaa79214104b151f1973a715db1e
Jun 19 16:41:27 dist-test-slave-03cd krb5kdc[4437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:41:27.988348 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:27.971818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35489 (local address 127.0.98.62:42321)
0619 16:41:27.971984 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:27.971989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:27.973018 (+ 1029us) server_negotiation.cc:407] Connection header received
0619 16:41:27.974825 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:27.974829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:27.974886 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:27.974977 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:27.976738 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:27.977636 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:27.979697 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:27.980122 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:27.980268 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:27.983802 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:27.983806 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:27.983808 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:27.983837 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:27.985634 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:27.986458 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:27.986462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:27.986464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:27.986565 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:27.986964 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:27.986967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:27.986969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:27.987180 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:27.987318 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:27.987975 (+ 657us) server_negotiation.cc:299] Negotiation successful
0619 16:41:27.988148 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 16:41:27.989703 4806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:27.972307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42321 (local address 127.0.98.2:35489)
0619 16:41:27.972839 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:27.972888 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:27.974485 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:27.975261 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:27.975272 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:27.975923 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:27.976559 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:27.976576 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:27.977841 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:27.977849 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:27.979496 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:27.979510 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:27.980336 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:27.980340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:27.980624 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:27.981444 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:27.981484 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:27.983525 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:27.985808 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:27.985815 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:27.985859 (+ 44us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:27.986346 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:27.986680 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:27.986683 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:27.986685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:27.986856 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:27.987301 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:27.987308 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:27.987644 (+ 336us) client_negotiation.cc:769] Sending connection context
0619 16:41:27.987921 (+ 277us) client_negotiation.cc:240] Negotiation successful
0619 16:41:27.988308 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":133,"threads_started":1}
I20260619 16:41:27.991060 4804 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42321
I20260619 16:41:27.991369 4804 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:27.991947 4804 heartbeater.cc:511] Master 127.0.98.62:42321 requested a full tablet report, sending...
I20260619 16:41:27.993209 4477 ts_manager.cc:194] Registered new tserver with Master: 27b1f952195c41dea8b1798cf105d0f0 (127.0.98.2:43639)
I20260619 16:41:27.994127 4477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:41:28.012311 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42321
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:28.139333 4809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:28.139758 4809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:28.139886 4809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:28.144512 4809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:28.144650 4809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:28.144767 4809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:41:28.150990 4809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:28.152396 4809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:28.153534 4809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:28.164168 4814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:28.164119 4817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:28.164304 4809 server_base.cc:1034] running on GCE node
W20260619 16:41:28.164126 4815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:28.164908 4809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:28.165683 4809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:28.166906 4809 hybrid_clock.cc:648] HybridClock initialized: now 1781887288166880 us; error 48 us; skew 500 ppm
Jun 19 16:41:28 dist-test-slave-03cd krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:28.170682 4809 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:41:28.171514 4809 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:28.171700 4809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:28.171959 4809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:28.174137 4809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-2/data/instance:
uuid: "5f8501aa25524b47a7b3c1a0adcfff84"
format_stamp: "Formatted at 2026-06-19 16:41:28 on dist-test-slave-03cd"
server_key: "65b686f07ecbe055da3c0ec19d3e261c"
server_key_iv: "b6ea7b62066a311c932af84cd30732b1"
server_key_version: "encryptionkey@0"
I20260619 16:41:28.174764 4809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5f8501aa25524b47a7b3c1a0adcfff84"
format_stamp: "Formatted at 2026-06-19 16:41:28 on dist-test-slave-03cd"
server_key: "65b686f07ecbe055da3c0ec19d3e261c"
server_key_iv: "b6ea7b62066a311c932af84cd30732b1"
server_key_version: "encryptionkey@0"
I20260619 16:41:28.178944 4809 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 16:41:28.181993 4823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:28.183466 4809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:41:28.183631 4809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "5f8501aa25524b47a7b3c1a0adcfff84"
format_stamp: "Formatted at 2026-06-19 16:41:28 on dist-test-slave-03cd"
server_key: "65b686f07ecbe055da3c0ec19d3e261c"
server_key_iv: "b6ea7b62066a311c932af84cd30732b1"
server_key_version: "encryptionkey@0"
I20260619 16:41:28.183853 4809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:28.218289 4809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:28.222239 4809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:28.222496 4809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:28.223151 4809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:28.224133 4809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:28.224213 4809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:28.224303 4809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:28.224352 4809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:28.234308 4809 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39947
I20260619 16:41:28.234326 4936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39947 every 8 connection(s)
I20260619 16:41:28.234979 4809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:41:28.240679 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 4809
I20260619 16:41:28.240837 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:41:28.241106 392 external_mini_cluster.cc:1428] Setting key 4f9cacda54e1ca7ff01624ebb7140c36
Jun 19 16:41:28 dist-test-slave-03cd krb5kdc[4437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:28.250639 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:28.236987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59571 (local address 127.0.98.62:42321)
0619 16:41:28.237371 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:28.237375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:28.238332 (+ 957us) server_negotiation.cc:407] Connection header received
0619 16:41:28.239739 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:28.239743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:28.239817 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:28.239900 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:28.241392 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:28.242041 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:28.243246 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:28.243521 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:28.243629 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:28.246357 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:28.246360 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:28.246362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:28.246387 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:28.248175 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:28.248757 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:28.248760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:28.248762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:28.248844 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:28.249388 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:28.249391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:28.249392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:28.249599 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:28.249695 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:28.250361 (+ 666us) server_negotiation.cc:299] Negotiation successful
0619 16:41:28.250494 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 16:41:28.251529 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:28.237660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42321 (local address 127.0.98.3:59571)
0619 16:41:28.238143 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:28.238188 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:28.239470 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:28.240068 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:28.240075 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:28.240534 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:28.241236 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:28.241252 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:28.242175 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:28.242178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:28.243109 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:28.243119 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:28.243728 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:28.243734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:28.243900 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:28.244516 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:28.244534 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:28.246117 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:28.248306 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:28.248312 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:28.248328 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:28.248629 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:28.248972 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:28.248975 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:28.248976 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:28.249283 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:28.249724 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:28.249730 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:28.249999 (+ 269us) client_negotiation.cc:769] Sending connection context
0619 16:41:28.250284 (+ 285us) client_negotiation.cc:240] Negotiation successful
0619 16:41:28.250589 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":119,"threads_started":1}
I20260619 16:41:28.253014 4937 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42321
I20260619 16:41:28.253289 4937 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:28.253881 4937 heartbeater.cc:511] Master 127.0.98.62:42321 requested a full tablet report, sending...
I20260619 16:41:28.255081 4477 ts_manager.cc:194] Registered new tserver with Master: 5f8501aa25524b47a7b3c1a0adcfff84 (127.0.98.3:39947)
I20260619 16:41:28.255729 4477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59571
I20260619 16:41:28.268038 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:41:28.280443 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:28.271028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47716 (local address 127.0.98.62:42321)
0619 16:41:28.271206 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:28.271214 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:28.271982 (+ 768us) server_negotiation.cc:407] Connection header received
0619 16:41:28.272128 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:28.272131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:28.272226 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:28.272319 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:28.273434 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:28.274178 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:28.275208 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:28.275528 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:28.275636 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:28.277075 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:28.277079 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:28.277082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:28.277122 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:28.278498 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:28.278993 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:28.278996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:28.278998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:28.279091 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:28.279379 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:28.279383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:28.279385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:28.279629 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:28.279818 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:28.280064 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 16:41:28.280185 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:41:28.285113 4477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:41:28.288509 4477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:41:28.305820 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:28.298626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60680 (local address 127.0.98.2:43639)
0619 16:41:28.299234 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:28.299247 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:28.299495 (+ 248us) server_negotiation.cc:407] Connection header received
0619 16:41:28.299786 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:28.299793 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:28.299994 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:28.300131 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:28.300782 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:28.302020 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:28.303950 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:28.304970 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:28.305107 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:28.305308 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:28.305492 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 16:41:28.305631 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":119,"threads_started":1}
I20260619 16:41:28.306413 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:28.298233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39947 (local address 127.0.0.1:36386)
0619 16:41:28.299155 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:28.299207 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:28.299425 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:28.300344 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:28.300350 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:28.300372 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:28.301922 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:28.301936 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:28.303350 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:28.303354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:28.304627 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:28.304639 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:28.305918 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:28.305922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:28.306019 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:28.306049 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 16:41:28.306146 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 16:41:28.306256 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":1360,"spinlock_wait_cycles":3328,"thread_start_us":115,"threads_started":1}
I20260619 16:41:28.306665 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:28.299056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46083 (local address 127.0.0.1:43204)
0619 16:41:28.299724 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:28.299737 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:28.299836 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:28.300440 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:28.300443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:28.300455 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:28.301924 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:28.301941 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:28.303538 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:28.303541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:28.304715 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:28.304726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:28.306152 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:28.306158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:28.306275 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:28.306301 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 16:41:28.306381 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 16:41:28.306462 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":1335,"thread_start_us":106,"threads_started":1}
I20260619 16:41:28.306926 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:28.298405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36386 (local address 127.0.98.3:39947)
0619 16:41:28.298894 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:28.298905 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:28.299303 (+ 398us) server_negotiation.cc:407] Connection header received
0619 16:41:28.299720 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:28.299725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:28.299972 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:28.300115 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:28.302104 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:28.303181 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:28.304804 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:28.305599 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:28.305714 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:28.305848 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:28.306613 (+ 765us) server_negotiation.cc:299] Negotiation successful
0619 16:41:28.306731 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":143,"threads_started":1}
I20260619 16:41:28.306985 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:28.298413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43639 (local address 127.0.0.1:60680)
0619 16:41:28.299397 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:28.299413 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:28.299577 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:28.300362 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:28.300365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:28.300393 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:28.300648 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:28.300656 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:28.302174 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:28.302178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:28.303817 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:28.303827 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:28.305124 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:28.305128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:28.305259 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:28.305307 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 16:41:28.305451 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 16:41:28.305600 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"mutex_wait_us":75,"spinlock_wait_cycles":768,"thread_start_us":109,"threads_started":1}
I20260619 16:41:28.307144 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:28.299221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43204 (local address 127.0.98.1:46083)
0619 16:41:28.299521 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:28.299529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:28.299926 (+ 397us) server_negotiation.cc:407] Connection header received
0619 16:41:28.300003 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:28.300008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:28.300178 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:28.300315 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:28.302067 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:28.303403 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:28.304885 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:28.305778 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:28.305927 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:28.306081 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:28.306847 (+ 766us) server_negotiation.cc:299] Negotiation successful
0619 16:41:28.306992 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":80,"threads_started":1}
I20260619 16:41:28.310143 4605 tablet_service.cc:1467] Processing CreateTablet for tablet bc1037076d554bc0b9b87f653c4d203c (DEFAULT_TABLE table=test-table [id=d900b6ee351a416ea16ec75808b4afb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:28.310148 4738 tablet_service.cc:1467] Processing CreateTablet for tablet bc1037076d554bc0b9b87f653c4d203c (DEFAULT_TABLE table=test-table [id=d900b6ee351a416ea16ec75808b4afb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:28.311363 4605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc1037076d554bc0b9b87f653c4d203c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:28.311363 4738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc1037076d554bc0b9b87f653c4d203c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:28.314114 4871 tablet_service.cc:1467] Processing CreateTablet for tablet bc1037076d554bc0b9b87f653c4d203c (DEFAULT_TABLE table=test-table [id=d900b6ee351a416ea16ec75808b4afb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:28.315402 4871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc1037076d554bc0b9b87f653c4d203c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:28.318034 4954 tablet_bootstrap.cc:492] T bc1037076d554bc0b9b87f653c4d203c P d4d9806468a24073998d0ab016c5fad6: Bootstrap starting.
I20260619 16:41:28.320356 4954 tablet_bootstrap.cc:654] T bc1037076d554bc0b9b87f653c4d203c P d4d9806468a24073998d0ab016c5fad6: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:28.321138 4954 log.cc:824] T bc1037076d554bc0b9b87f653c4d203c P d4d9806468a24073998d0ab016c5fad6: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:28.323355 4957 tablet_bootstrap.cc:492] T bc1037076d554bc0b9b87f653c4d203c P 5f8501aa25524b47a7b3c1a0adcfff84: Bootstrap starting.
I20260619 16:41:28.324038 4955 tablet_bootstrap.cc:492] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0: Bootstrap starting.
I20260619 16:41:28.325484 4957 tablet_bootstrap.cc:654] T bc1037076d554bc0b9b87f653c4d203c P 5f8501aa25524b47a7b3c1a0adcfff84: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:28.325709 4955 tablet_bootstrap.cc:654] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:28.326687 4957 log.cc:824] T bc1037076d554bc0b9b87f653c4d203c P 5f8501aa25524b47a7b3c1a0adcfff84: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:28.326865 4955 log.cc:824] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:28.327642 4954 tablet_bootstrap.cc:492] T bc1037076d554bc0b9b87f653c4d203c P d4d9806468a24073998d0ab016c5fad6: No bootstrap required, opened a new log
I20260619 16:41:28.327920 4954 ts_tablet_manager.cc:1397] T bc1037076d554bc0b9b87f653c4d203c P d4d9806468a24073998d0ab016c5fad6: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.002s
I20260619 16:41:28.328737 4955 tablet_bootstrap.cc:492] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0: No bootstrap required, opened a new log
I20260619 16:41:28.328758 4957 tablet_bootstrap.cc:492] T bc1037076d554bc0b9b87f653c4d203c P 5f8501aa25524b47a7b3c1a0adcfff84: No bootstrap required, opened a new log
I20260619 16:41:28.328940 4957 ts_tablet_manager.cc:1397] T bc1037076d554bc0b9b87f653c4d203c P 5f8501aa25524b47a7b3c1a0adcfff84: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 16:41:28.328958 4955 ts_tablet_manager.cc:1397] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260619 16:41:28.332024 4955 raft_consensus.cc:348] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8501aa25524b47a7b3c1a0adcfff84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39947 } } peers { permanent_uuid: "d4d9806468a24073998d0ab016c5fad6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46083 } } peers { permanent_uuid: "27b1f952195c41dea8b1798cf105d0f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43639 } }
I20260619 16:41:28.332311 4955 raft_consensus.cc:374] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:28.332401 4955 raft_consensus.cc:729] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27b1f952195c41dea8b1798cf105d0f0, State: Initialized, Role: FOLLOWER
I20260619 16:41:28.332491 4954 raft_consensus.cc:348] T bc1037076d554bc0b9b87f653c4d203c P d4d9806468a24073998d0ab016c5fad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8501aa25524b47a7b3c1a0adcfff84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39947 } } peers { permanent_uuid: "d4d9806468a24073998d0ab016c5fad6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46083 } } peers { permanent_uuid: "27b1f952195c41dea8b1798cf105d0f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43639 } }
I20260619 16:41:28.332747 4954 raft_consensus.cc:374] T bc1037076d554bc0b9b87f653c4d203c P d4d9806468a24073998d0ab016c5fad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:28.332842 4954 raft_consensus.cc:729] T bc1037076d554bc0b9b87f653c4d203c P d4d9806468a24073998d0ab016c5fad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4d9806468a24073998d0ab016c5fad6, State: Initialized, Role: FOLLOWER
I20260619 16:41:28.332720 4957 raft_consensus.cc:348] T bc1037076d554bc0b9b87f653c4d203c P 5f8501aa25524b47a7b3c1a0adcfff84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8501aa25524b47a7b3c1a0adcfff84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39947 } } peers { permanent_uuid: "d4d9806468a24073998d0ab016c5fad6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46083 } } peers { permanent_uuid: "27b1f952195c41dea8b1798cf105d0f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43639 } }
I20260619 16:41:28.332986 4957 raft_consensus.cc:374] T bc1037076d554bc0b9b87f653c4d203c P 5f8501aa25524b47a7b3c1a0adcfff84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:28.333042 4957 raft_consensus.cc:729] T bc1037076d554bc0b9b87f653c4d203c P 5f8501aa25524b47a7b3c1a0adcfff84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8501aa25524b47a7b3c1a0adcfff84, State: Initialized, Role: FOLLOWER
I20260619 16:41:28.332954 4955 consensus_queue.cc:260] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8501aa25524b47a7b3c1a0adcfff84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39947 } } peers { permanent_uuid: "d4d9806468a24073998d0ab016c5fad6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46083 } } peers { permanent_uuid: "27b1f952195c41dea8b1798cf105d0f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43639 } }
I20260619 16:41:28.333266 4954 consensus_queue.cc:260] T bc1037076d554bc0b9b87f653c4d203c P d4d9806468a24073998d0ab016c5fad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8501aa25524b47a7b3c1a0adcfff84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39947 } } peers { permanent_uuid: "d4d9806468a24073998d0ab016c5fad6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46083 } } peers { permanent_uuid: "27b1f952195c41dea8b1798cf105d0f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43639 } }
I20260619 16:41:28.333422 4957 consensus_queue.cc:260] T bc1037076d554bc0b9b87f653c4d203c P 5f8501aa25524b47a7b3c1a0adcfff84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8501aa25524b47a7b3c1a0adcfff84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39947 } } peers { permanent_uuid: "d4d9806468a24073998d0ab016c5fad6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46083 } } peers { permanent_uuid: "27b1f952195c41dea8b1798cf105d0f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43639 } }
I20260619 16:41:28.333791 4804 heartbeater.cc:503] Master 127.0.98.62:42321 was elected leader, sending a full tablet report...
I20260619 16:41:28.334122 4955 ts_tablet_manager.cc:1428] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260619 16:41:28.334245 4954 ts_tablet_manager.cc:1428] T bc1037076d554bc0b9b87f653c4d203c P d4d9806468a24073998d0ab016c5fad6: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260619 16:41:28.334245 4957 ts_tablet_manager.cc:1428] T bc1037076d554bc0b9b87f653c4d203c P 5f8501aa25524b47a7b3c1a0adcfff84: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 16:41:28.334568 4671 heartbeater.cc:503] Master 127.0.98.62:42321 was elected leader, sending a full tablet report...
I20260619 16:41:28.334578 4937 heartbeater.cc:503] Master 127.0.98.62:42321 was elected leader, sending a full tablet report...
W20260619 16:41:28.471163 4805 tablet.cc:2369] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:41:28.485967 4938 tablet.cc:2369] T bc1037076d554bc0b9b87f653c4d203c P 5f8501aa25524b47a7b3c1a0adcfff84: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:41:28.493866 4672 tablet.cc:2369] T bc1037076d554bc0b9b87f653c4d203c P d4d9806468a24073998d0ab016c5fad6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:41:28.589390 4960 raft_consensus.cc:482] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:41:28.589654 4960 raft_consensus.cc:504] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8501aa25524b47a7b3c1a0adcfff84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39947 } } peers { permanent_uuid: "d4d9806468a24073998d0ab016c5fad6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46083 } } peers { permanent_uuid: "27b1f952195c41dea8b1798cf105d0f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43639 } }
I20260619 16:41:28.591094 4960 leader_election.cc:288] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8501aa25524b47a7b3c1a0adcfff84 (127.0.98.3:39947), d4d9806468a24073998d0ab016c5fad6 (127.0.98.1:46083)
I20260619 16:41:28.595474 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:28.591489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32925 (local address 127.0.98.3:39947)
0619 16:41:28.591707 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:28.591713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:28.591733 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:41:28.591917 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:28.591923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:28.591985 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:28.592105 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:28.592432 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:28.592948 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:28.594472 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:28.594903 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:28.595028 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:28.595096 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:28.595272 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 16:41:28.595349 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 16:41:28.595494 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:28.591312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39947 (local address 127.0.98.2:32925)
0619 16:41:28.591604 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:28.591622 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:28.591757 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:28.592121 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:28.592125 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:28.592175 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:28.592306 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:28.592313 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:28.593066 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:28.593073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:28.594305 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:28.594318 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:28.595030 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:28.595033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:28.595131 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:28.595155 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 16:41:28.595256 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 16:41:28.595333 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"spinlock_wait_cycles":40448,"thread_start_us":112,"threads_started":1}
I20260619 16:41:28.595752 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:28.591903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46143 (local address 127.0.98.1:46083)
0619 16:41:28.592048 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:28.592053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:28.592128 (+ 75us) server_negotiation.cc:407] Connection header received
0619 16:41:28.592288 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:28.592291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:28.592342 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:28.592431 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:28.592688 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:28.593278 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:28.594561 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:28.595122 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:28.595270 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:28.595346 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:28.595547 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 16:41:28.595620 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:41:28.595767 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:28.591756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46083 (local address 127.0.98.2:46143)
0619 16:41:28.592017 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:28.592029 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:28.592183 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:28.592441 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:28.592443 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:28.592458 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:28.592584 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:28.592593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:28.593404 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:28.593410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:28.594433 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:28.594444 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:28.595271 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:28.595275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:28.595396 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:28.595425 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 16:41:28.595516 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 16:41:28.595592 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"spinlock_wait_cycles":7296,"thread_start_us":90,"threads_started":1}
I20260619 16:41:28.596597 4891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1037076d554bc0b9b87f653c4d203c" candidate_uuid: "27b1f952195c41dea8b1798cf105d0f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8501aa25524b47a7b3c1a0adcfff84" is_pre_election: true
I20260619 16:41:28.596594 4625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1037076d554bc0b9b87f653c4d203c" candidate_uuid: "27b1f952195c41dea8b1798cf105d0f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4d9806468a24073998d0ab016c5fad6" is_pre_election: true
I20260619 16:41:28.596956 4625 raft_consensus.cc:2454] T bc1037076d554bc0b9b87f653c4d203c P d4d9806468a24073998d0ab016c5fad6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27b1f952195c41dea8b1798cf105d0f0 in term 0.
I20260619 16:41:28.596956 4891 raft_consensus.cc:2454] T bc1037076d554bc0b9b87f653c4d203c P 5f8501aa25524b47a7b3c1a0adcfff84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27b1f952195c41dea8b1798cf105d0f0 in term 0.
I20260619 16:41:28.597467 4691 leader_election.cc:302] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27b1f952195c41dea8b1798cf105d0f0, d4d9806468a24073998d0ab016c5fad6; no voters:
I20260619 16:41:28.597731 4960 raft_consensus.cc:2781] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:41:28.597818 4960 raft_consensus.cc:482] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:41:28.597851 4960 raft_consensus.cc:3037] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:28.598987 4960 raft_consensus.cc:504] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8501aa25524b47a7b3c1a0adcfff84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39947 } } peers { permanent_uuid: "d4d9806468a24073998d0ab016c5fad6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46083 } } peers { permanent_uuid: "27b1f952195c41dea8b1798cf105d0f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43639 } }
I20260619 16:41:28.599480 4960 leader_election.cc:288] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8501aa25524b47a7b3c1a0adcfff84 (127.0.98.3:39947), d4d9806468a24073998d0ab016c5fad6 (127.0.98.1:46083)
I20260619 16:41:28.600104 4625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1037076d554bc0b9b87f653c4d203c" candidate_uuid: "27b1f952195c41dea8b1798cf105d0f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4d9806468a24073998d0ab016c5fad6"
I20260619 16:41:28.600106 4891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1037076d554bc0b9b87f653c4d203c" candidate_uuid: "27b1f952195c41dea8b1798cf105d0f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8501aa25524b47a7b3c1a0adcfff84"
I20260619 16:41:28.600255 4891 raft_consensus.cc:3037] T bc1037076d554bc0b9b87f653c4d203c P 5f8501aa25524b47a7b3c1a0adcfff84 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:28.600255 4625 raft_consensus.cc:3037] T bc1037076d554bc0b9b87f653c4d203c P d4d9806468a24073998d0ab016c5fad6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:28.601598 4625 raft_consensus.cc:2454] T bc1037076d554bc0b9b87f653c4d203c P d4d9806468a24073998d0ab016c5fad6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27b1f952195c41dea8b1798cf105d0f0 in term 1.
I20260619 16:41:28.601611 4891 raft_consensus.cc:2454] T bc1037076d554bc0b9b87f653c4d203c P 5f8501aa25524b47a7b3c1a0adcfff84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27b1f952195c41dea8b1798cf105d0f0 in term 1.
I20260619 16:41:28.601984 4691 leader_election.cc:302] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27b1f952195c41dea8b1798cf105d0f0, d4d9806468a24073998d0ab016c5fad6; no voters:
I20260619 16:41:28.602272 4960 raft_consensus.cc:2781] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:28.602568 4960 raft_consensus.cc:686] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0 [term 1 LEADER]: Becoming Leader. State: Replica: 27b1f952195c41dea8b1798cf105d0f0, State: Running, Role: LEADER
I20260619 16:41:28.602964 4960 consensus_queue.cc:237] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8501aa25524b47a7b3c1a0adcfff84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39947 } } peers { permanent_uuid: "d4d9806468a24073998d0ab016c5fad6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46083 } } peers { permanent_uuid: "27b1f952195c41dea8b1798cf105d0f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43639 } }
I20260619 16:41:28.606374 4477 catalog_manager.cc:5495] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27b1f952195c41dea8b1798cf105d0f0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "27b1f952195c41dea8b1798cf105d0f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8501aa25524b47a7b3c1a0adcfff84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4d9806468a24073998d0ab016c5fad6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27b1f952195c41dea8b1798cf105d0f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43639 } health_report { overall_health: HEALTHY } } }
I20260619 16:41:28.674057 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:28.670422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60690 (local address 127.0.98.2:43639)
0619 16:41:28.670579 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:28.670588 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:28.670600 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:41:28.670671 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:28.670673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:28.670726 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:28.670842 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:41:28.671106 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:28.671638 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:28.672524 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:28.672749 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:28.672866 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:28.673194 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:41:28.673638 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:41:28.673778 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:28.673839 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 16:41:28.673901 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:41:28.680377 4891 raft_consensus.cc:1261] T bc1037076d554bc0b9b87f653c4d203c P 5f8501aa25524b47a7b3c1a0adcfff84 [term 1 FOLLOWER]: Refusing update from remote peer 27b1f952195c41dea8b1798cf105d0f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:28.680375 4625 raft_consensus.cc:1261] T bc1037076d554bc0b9b87f653c4d203c P d4d9806468a24073998d0ab016c5fad6 [term 1 FOLLOWER]: Refusing update from remote peer 27b1f952195c41dea8b1798cf105d0f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:28.681201 4960 consensus_queue.cc:1035] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8501aa25524b47a7b3c1a0adcfff84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:28.681598 4965 consensus_queue.cc:1035] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4d9806468a24073998d0ab016c5fad6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:28.689587 4968 mvcc.cc:205] Tried to move back new op lower bound from 7298610334425968640 to 7298610334120484864. Current Snapshot: MvccSnapshot[applied={T|T < 7298610334425968640}]
I20260619 16:41:28.690194 4969 mvcc.cc:205] Tried to move back new op lower bound from 7298610334425968640 to 7298610334120484864. Current Snapshot: MvccSnapshot[applied={T|T < 7298610334425968640}]
I20260619 16:41:28.699563 4477 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:41:28.699788 4477 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:41:28.702778 4477 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7b0c7066b564a58b368550fef7e45e3: Sending DeleteTablet for 3 replicas of tablet bc1037076d554bc0b9b87f653c4d203c
I20260619 16:41:28.703855 4738 tablet_service.cc:1514] Processing DeleteTablet for tablet bc1037076d554bc0b9b87f653c4d203c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60680
I20260619 16:41:28.703922 4605 tablet_service.cc:1514] Processing DeleteTablet for tablet bc1037076d554bc0b9b87f653c4d203c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43204
I20260619 16:41:28.704105 4871 tablet_service.cc:1514] Processing DeleteTablet for tablet bc1037076d554bc0b9b87f653c4d203c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36386
I20260619 16:41:28.704617 4978 tablet_replica.cc:330] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0: stopping tablet replica
I20260619 16:41:28.704618 4980 tablet_replica.cc:330] T bc1037076d554bc0b9b87f653c4d203c P 5f8501aa25524b47a7b3c1a0adcfff84: stopping tablet replica
I20260619 16:41:28.704979 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 4543
I20260619 16:41:28.705010 4978 raft_consensus.cc:2229] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:41:28.705010 4980 raft_consensus.cc:2229] T bc1037076d554bc0b9b87f653c4d203c P 5f8501aa25524b47a7b3c1a0adcfff84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:41:28.705374 4980 raft_consensus.cc:2258] T bc1037076d554bc0b9b87f653c4d203c P 5f8501aa25524b47a7b3c1a0adcfff84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:28.705384 4978 raft_consensus.cc:2258] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:28.707420 4978 ts_tablet_manager.cc:1905] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:28.707472 4980 ts_tablet_manager.cc:1905] T bc1037076d554bc0b9b87f653c4d203c P 5f8501aa25524b47a7b3c1a0adcfff84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:28.710392 4978 ts_tablet_manager.cc:1918] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:41:28.710505 4978 log.cc:1196] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-1/wal/wals/bc1037076d554bc0b9b87f653c4d203c
I20260619 16:41:28.710700 4980 ts_tablet_manager.cc:1918] T bc1037076d554bc0b9b87f653c4d203c P 5f8501aa25524b47a7b3c1a0adcfff84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:41:28.710819 4980 log.cc:1196] T bc1037076d554bc0b9b87f653c4d203c P 5f8501aa25524b47a7b3c1a0adcfff84: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887259574980-392-0/minicluster-data/ts-2/wal/wals/bc1037076d554bc0b9b87f653c4d203c
I20260619 16:41:28.710820 4978 ts_tablet_manager.cc:1939] T bc1037076d554bc0b9b87f653c4d203c P 27b1f952195c41dea8b1798cf105d0f0: Deleting consensus metadata
I20260619 16:41:28.711139 4980 ts_tablet_manager.cc:1939] T bc1037076d554bc0b9b87f653c4d203c P 5f8501aa25524b47a7b3c1a0adcfff84: Deleting consensus metadata
I20260619 16:41:28.712240 4467 catalog_manager.cc:4841] TS 27b1f952195c41dea8b1798cf105d0f0 (127.0.98.2:43639): tablet bc1037076d554bc0b9b87f653c4d203c (table test-table [id=d900b6ee351a416ea16ec75808b4afb2]) successfully deleted
I20260619 16:41:28.712419 4465 catalog_manager.cc:4841] TS 5f8501aa25524b47a7b3c1a0adcfff84 (127.0.98.3:39947): tablet bc1037076d554bc0b9b87f653c4d203c (table test-table [id=d900b6ee351a416ea16ec75808b4afb2]) successfully deleted
W20260619 16:41:28.716642 4465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46083 (error 108)
I20260619 16:41:28.717669 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 4676
I20260619 16:41:28.717923 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:28.717554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46083 (local address 127.0.0.1:43216)
0619 16:41:28.717726 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:28.717810 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 16:41:28.719939 4465 catalog_manager.cc:4584] TS d4d9806468a24073998d0ab016c5fad6 (127.0.98.1:46083): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bc1037076d554bc0b9b87f653c4d203c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46083: connect: Connection refused (error 111)
I20260619 16:41:28.724226 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 4809
I20260619 16:41:28.730370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 4450
2026-06-19T16:41:28Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2779 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T16:41:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:41:28Z Disabled control of system clock
I20260619 16:41:28.764082 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42343
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 16:41:28.885516 4985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:28.885886 4985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:28.885927 4985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:28.891419 4985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:41:28.891527 4985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:28.891548 4985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:28.891569 4985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:41:28.891589 4985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:41:28.897346 4985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42343
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:28.898530 4985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:28.900027 4985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:28.910027 4993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:28.910127 4991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:28.910115 4990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:28.910712 4985 server_base.cc:1034] running on GCE node
I20260619 16:41:28.911514 4985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:28.913086 4985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:28.914296 4985 hybrid_clock.cc:648] HybridClock initialized: now 1781887288914254 us; error 63 us; skew 500 ppm
I20260619 16:41:28.917019 4985 webserver.cc:460] Webserver started at http://127.0.98.62:42039/ using document root <none> and password file <none>
I20260619 16:41:28.917824 4985 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:28.917917 4985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:28.918208 4985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:28.921030 4985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "6778d1f8056741e289bf327ea6d1e8d0"
format_stamp: "Formatted at 2026-06-19 16:41:28 on dist-test-slave-03cd"
server_key: "3361b8fe04c4ad520032021d6ecf14a0"
server_key_iv: "f0953f3a2a147e4a5282a1f004d74be3"
server_key_version: "encryptionkey@0"
I20260619 16:41:28.921825 4985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "6778d1f8056741e289bf327ea6d1e8d0"
format_stamp: "Formatted at 2026-06-19 16:41:28 on dist-test-slave-03cd"
server_key: "3361b8fe04c4ad520032021d6ecf14a0"
server_key_iv: "f0953f3a2a147e4a5282a1f004d74be3"
server_key_version: "encryptionkey@0"
I20260619 16:41:28.927593 4985 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:41:28.931792 4999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:28.933897 4985 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:41:28.934161 4985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "6778d1f8056741e289bf327ea6d1e8d0"
format_stamp: "Formatted at 2026-06-19 16:41:28 on dist-test-slave-03cd"
server_key: "3361b8fe04c4ad520032021d6ecf14a0"
server_key_iv: "f0953f3a2a147e4a5282a1f004d74be3"
server_key_version: "encryptionkey@0"
I20260619 16:41:28.934424 4985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:28.941164 4985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:28.942104 4985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:28.942348 4985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:28.951730 4985 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42343
I20260619 16:41:28.951804 5051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42343 every 8 connection(s)
I20260619 16:41:28.953086 4985 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:41:28.956842 5052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:28.962148 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 4985
I20260619 16:41:28.962352 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:41:28.962706 392 external_mini_cluster.cc:1428] Setting key 194b92d42eee87782a18283744e53e8a
I20260619 16:41:28.963455 5052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6778d1f8056741e289bf327ea6d1e8d0: Bootstrap starting.
I20260619 16:41:28.966804 5052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6778d1f8056741e289bf327ea6d1e8d0: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:28.967864 5052 log.cc:824] T 00000000000000000000000000000000 P 6778d1f8056741e289bf327ea6d1e8d0: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:28.970701 5052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6778d1f8056741e289bf327ea6d1e8d0: No bootstrap required, opened a new log
I20260619 16:41:28.971158 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:28.964931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51154 (local address 127.0.98.62:42343)
0619 16:41:28.965938 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:28.965961 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:28.966014 (+ 53us) server_negotiation.cc:407] Connection header received
0619 16:41:28.967078 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:28.967106 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:28.967655 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:28.968293 (+ 638us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:28.969373 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:28.969379 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:28.969395 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:41:28.969407 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:28.969479 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:41:28.969501 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:28.969658 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:28.969751 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 16:41:28.970104 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":186,"threads_started":1}
I20260619 16:41:28.975901 5052 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6778d1f8056741e289bf327ea6d1e8d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6778d1f8056741e289bf327ea6d1e8d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } }
I20260619 16:41:28.976195 5052 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6778d1f8056741e289bf327ea6d1e8d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:28.976272 5052 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6778d1f8056741e289bf327ea6d1e8d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6778d1f8056741e289bf327ea6d1e8d0, State: Initialized, Role: FOLLOWER
I20260619 16:41:28.976819 5052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6778d1f8056741e289bf327ea6d1e8d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6778d1f8056741e289bf327ea6d1e8d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } }
I20260619 16:41:28.976958 5052 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6778d1f8056741e289bf327ea6d1e8d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:41:28.977059 5052 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6778d1f8056741e289bf327ea6d1e8d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:41:28.977188 5052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6778d1f8056741e289bf327ea6d1e8d0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:28.978456 5052 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6778d1f8056741e289bf327ea6d1e8d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6778d1f8056741e289bf327ea6d1e8d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } }
I20260619 16:41:28.978885 5052 leader_election.cc:302] T 00000000000000000000000000000000 P 6778d1f8056741e289bf327ea6d1e8d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6778d1f8056741e289bf327ea6d1e8d0; no voters:
I20260619 16:41:28.979265 5052 leader_election.cc:288] T 00000000000000000000000000000000 P 6778d1f8056741e289bf327ea6d1e8d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:41:28.979349 5057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6778d1f8056741e289bf327ea6d1e8d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:28.979569 5057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6778d1f8056741e289bf327ea6d1e8d0 [term 1 LEADER]: Becoming Leader. State: Replica: 6778d1f8056741e289bf327ea6d1e8d0, State: Running, Role: LEADER
I20260619 16:41:28.980358 5057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6778d1f8056741e289bf327ea6d1e8d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6778d1f8056741e289bf327ea6d1e8d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } }
I20260619 16:41:28.980804 5052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6778d1f8056741e289bf327ea6d1e8d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:41:28.982250 5058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6778d1f8056741e289bf327ea6d1e8d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6778d1f8056741e289bf327ea6d1e8d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6778d1f8056741e289bf327ea6d1e8d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } } }
I20260619 16:41:28.982402 5058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6778d1f8056741e289bf327ea6d1e8d0 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:28.982740 5059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6778d1f8056741e289bf327ea6d1e8d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6778d1f8056741e289bf327ea6d1e8d0. Latest consensus state: current_term: 1 leader_uuid: "6778d1f8056741e289bf327ea6d1e8d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6778d1f8056741e289bf327ea6d1e8d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } } }
I20260619 16:41:28.982864 5059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6778d1f8056741e289bf327ea6d1e8d0 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:28.983048 5067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:41:28.986908 5067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:41:28.994772 5067 catalog_manager.cc:1347] Generated new cluster ID: 834ad31903ab41cc9abbcb03ccad61c0
I20260619 16:41:28.994899 5067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:41:29.009322 5067 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:41:29.009809 5067 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:41:29.017037 5067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6778d1f8056741e289bf327ea6d1e8d0: Generated new TSK 0
I20260619 16:41:29.018087 5067 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:41:29.023072 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42343
--builtin_ntp_servers=127.0.98.20:43949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 16:41:29.167951 5076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:29.168344 5076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:29.168396 5076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:29.173076 5076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:29.173178 5076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:29.173249 5076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:41:29.178771 5076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:29.180511 5076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:29.182543 5076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:29.196272 5081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:29.196300 5082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:29.196435 5076 server_base.cc:1034] running on GCE node
W20260619 16:41:29.196260 5084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:29.197196 5076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:29.198236 5076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:29.199509 5076 hybrid_clock.cc:648] HybridClock initialized: now 1781887289199472 us; error 49 us; skew 500 ppm
I20260619 16:41:29.201901 5076 webserver.cc:460] Webserver started at http://127.0.98.1:37061/ using document root <none> and password file <none>
I20260619 16:41:29.202625 5076 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:29.202731 5076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:29.203006 5076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:29.205431 5076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-0/data/instance:
uuid: "7bc89a43b3404eaa83341b68be1c75bf"
format_stamp: "Formatted at 2026-06-19 16:41:29 on dist-test-slave-03cd"
server_key: "debc86f52bd27f924342da3959874e75"
server_key_iv: "2af0f7085514b81c894095c06c29b4eb"
server_key_version: "encryptionkey@0"
I20260619 16:41:29.206096 5076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7bc89a43b3404eaa83341b68be1c75bf"
format_stamp: "Formatted at 2026-06-19 16:41:29 on dist-test-slave-03cd"
server_key: "debc86f52bd27f924342da3959874e75"
server_key_iv: "2af0f7085514b81c894095c06c29b4eb"
server_key_version: "encryptionkey@0"
I20260619 16:41:29.211102 5076 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:41:29.214531 5090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:29.216315 5076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 16:41:29.216502 5076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "7bc89a43b3404eaa83341b68be1c75bf"
format_stamp: "Formatted at 2026-06-19 16:41:29 on dist-test-slave-03cd"
server_key: "debc86f52bd27f924342da3959874e75"
server_key_iv: "2af0f7085514b81c894095c06c29b4eb"
server_key_version: "encryptionkey@0"
I20260619 16:41:29.216706 5076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:29.222328 5076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:29.223068 5076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:29.223297 5076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:29.224022 5076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:29.225143 5076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:29.225225 5076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:29.225303 5076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:29.225343 5076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:29.236744 5076 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35695
I20260619 16:41:29.236764 5203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35695 every 8 connection(s)
I20260619 16:41:29.237859 5076 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:41:29.241966 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 5076
I20260619 16:41:29.242133 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:41:29.242509 392 external_mini_cluster.cc:1428] Setting key f496acdf01f855b86968f01373ad645f
I20260619 16:41:29.245254 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42343
--builtin_ntp_servers=127.0.98.20:43949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 16:41:29.247970 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:29.240899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35141 (local address 127.0.98.62:42343)
0619 16:41:29.241109 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:29.241115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:29.243005 (+ 1890us) server_negotiation.cc:407] Connection header received
0619 16:41:29.245201 (+ 2196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:29.245205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:29.245260 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:29.245338 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:29.247210 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:29.247218 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:29.247223 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:41:29.247248 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:29.247287 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:41:29.247304 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:29.247449 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:29.247622 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 16:41:29.247733 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 16:41:29.249002 5206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:29.241474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.1:35141)
0619 16:41:29.242725 (+ 1251us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:29.242790 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:29.244859 (+ 2069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:29.245689 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:29.245700 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:29.245853 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:29.246770 (+ 917us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:41:29.246772 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:29.246839 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:41:29.246957 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:29.247437 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:29.247445 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:29.247450 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:41:29.247720 (+ 270us) client_negotiation.cc:240] Negotiation successful
0619 16:41:29.248007 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":216,"threads_started":1}
I20260619 16:41:29.250722 5204 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42343
I20260619 16:41:29.251209 5204 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:29.252128 5204 heartbeater.cc:511] Master 127.0.98.62:42343 requested a full tablet report, sending...
I20260619 16:41:29.254606 5016 ts_manager.cc:194] Registered new tserver with Master: 7bc89a43b3404eaa83341b68be1c75bf (127.0.98.1:35695)
W20260619 16:41:29.394937 5207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:29.395340 5207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:29.395396 5207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:29.400074 5207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:29.400177 5207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:29.400249 5207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:41:29.405766 5207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:29.407311 5207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:29.409348 5207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:29.422115 5213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:29.422058 5215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:29.422317 5212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:29.422478 5207 server_base.cc:1034] running on GCE node
I20260619 16:41:29.423193 5207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:29.424212 5207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:29.425424 5207 hybrid_clock.cc:648] HybridClock initialized: now 1781887289425378 us; error 65 us; skew 500 ppm
I20260619 16:41:29.428285 5207 webserver.cc:460] Webserver started at http://127.0.98.2:35023/ using document root <none> and password file <none>
I20260619 16:41:29.429105 5207 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:29.429189 5207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:29.429450 5207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:29.432159 5207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-1/data/instance:
uuid: "3f6411f2a9e44486a207358821ad3de7"
format_stamp: "Formatted at 2026-06-19 16:41:29 on dist-test-slave-03cd"
server_key: "f3b6dc649a94a31378e295325007718a"
server_key_iv: "9c5027a3f1a8ec1d2ac99490d437bf09"
server_key_version: "encryptionkey@0"
I20260619 16:41:29.432852 5207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3f6411f2a9e44486a207358821ad3de7"
format_stamp: "Formatted at 2026-06-19 16:41:29 on dist-test-slave-03cd"
server_key: "f3b6dc649a94a31378e295325007718a"
server_key_iv: "9c5027a3f1a8ec1d2ac99490d437bf09"
server_key_version: "encryptionkey@0"
I20260619 16:41:29.437969 5207 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:41:29.442092 5221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:29.444169 5207 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:41:29.444360 5207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "3f6411f2a9e44486a207358821ad3de7"
format_stamp: "Formatted at 2026-06-19 16:41:29 on dist-test-slave-03cd"
server_key: "f3b6dc649a94a31378e295325007718a"
server_key_iv: "9c5027a3f1a8ec1d2ac99490d437bf09"
server_key_version: "encryptionkey@0"
I20260619 16:41:29.444665 5207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:29.451221 5207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:29.452217 5207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:29.452483 5207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:29.453258 5207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:29.454545 5207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:29.454605 5207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:29.454654 5207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:29.454670 5207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:29.467221 5207 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44071
I20260619 16:41:29.467329 5334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44071 every 8 connection(s)
I20260619 16:41:29.468593 5207 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:41:29.474888 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 5207
I20260619 16:41:29.475071 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:41:29.475399 392 external_mini_cluster.cc:1428] Setting key d99cf64eb0be893952c8bf187a2d5ba0
I20260619 16:41:29.477092 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:29.471080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33039 (local address 127.0.98.62:42343)
0619 16:41:29.471236 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:29.471240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:29.472864 (+ 1624us) server_negotiation.cc:407] Connection header received
0619 16:41:29.474353 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:29.474357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:29.474415 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:29.474544 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:29.476474 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:29.476481 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:29.476485 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:41:29.476502 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:29.476542 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:41:29.476553 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:29.476655 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:29.476889 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 16:41:29.476951 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:41:29.478214 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:29.471610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.2:33039)
0619 16:41:29.472635 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:29.472692 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:29.474054 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:29.474863 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:29.474876 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:29.475026 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:29.476042 (+ 1016us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:41:29.476044 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:29.476109 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:41:29.476216 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:29.476680 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:29.476689 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:29.476695 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:41:29.476913 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 16:41:29.477263 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":143,"threads_started":1}
I20260619 16:41:29.478372 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42343
--builtin_ntp_servers=127.0.98.20:43949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 16:41:29.479964 5335 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42343
I20260619 16:41:29.480424 5335 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:29.481089 5335 heartbeater.cc:511] Master 127.0.98.62:42343 requested a full tablet report, sending...
I20260619 16:41:29.482822 5016 ts_manager.cc:194] Registered new tserver with Master: 3f6411f2a9e44486a207358821ad3de7 (127.0.98.2:44071)
W20260619 16:41:29.627928 5338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:29.628438 5338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:29.628504 5338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:29.635620 5338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:29.635790 5338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:29.635939 5338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:41:29.647830 5338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:29.650034 5338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:29.652689 5338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:29.665669 5346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:29.665696 5344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:29.665709 5343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:29.666028 5338 server_base.cc:1034] running on GCE node
I20260619 16:41:29.666584 5338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:29.667416 5338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:29.668673 5338 hybrid_clock.cc:648] HybridClock initialized: now 1781887289668644 us; error 55 us; skew 500 ppm
I20260619 16:41:29.671109 5338 webserver.cc:460] Webserver started at http://127.0.98.3:33651/ using document root <none> and password file <none>
I20260619 16:41:29.671923 5338 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:29.672029 5338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:29.672312 5338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:29.674746 5338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-2/data/instance:
uuid: "7000338d8b5a48589b2c427e9b9290a5"
format_stamp: "Formatted at 2026-06-19 16:41:29 on dist-test-slave-03cd"
server_key: "7d469ede69773c9e87059da750b52a6d"
server_key_iv: "6d14dd0397976003185872732acd7218"
server_key_version: "encryptionkey@0"
I20260619 16:41:29.675436 5338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7000338d8b5a48589b2c427e9b9290a5"
format_stamp: "Formatted at 2026-06-19 16:41:29 on dist-test-slave-03cd"
server_key: "7d469ede69773c9e87059da750b52a6d"
server_key_iv: "6d14dd0397976003185872732acd7218"
server_key_version: "encryptionkey@0"
I20260619 16:41:29.680373 5338 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 16:41:29.683915 5352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:29.685727 5338 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:41:29.685992 5338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "7000338d8b5a48589b2c427e9b9290a5"
format_stamp: "Formatted at 2026-06-19 16:41:29 on dist-test-slave-03cd"
server_key: "7d469ede69773c9e87059da750b52a6d"
server_key_iv: "6d14dd0397976003185872732acd7218"
server_key_version: "encryptionkey@0"
I20260619 16:41:29.686161 5338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:29.691768 5338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:29.692489 5338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:29.692701 5338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:29.693346 5338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:29.694747 5338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:29.694828 5338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:29.694903 5338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:29.694958 5338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:29.705704 5338 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43347
I20260619 16:41:29.705798 5465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43347 every 8 connection(s)
I20260619 16:41:29.706790 5338 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:41:29.707945 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 5338
I20260619 16:41:29.708096 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:41:29.708381 392 external_mini_cluster.cc:1428] Setting key 576cb4f4435d16b4ad2fb78d7a9f0047
I20260619 16:41:29.714834 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:29.709274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36203 (local address 127.0.98.62:42343)
0619 16:41:29.709435 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:29.709442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:29.711275 (+ 1833us) server_negotiation.cc:407] Connection header received
0619 16:41:29.712742 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:29.712745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:29.712792 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:29.712876 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:29.714341 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:29.714345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:29.714348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:41:29.714366 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:29.714386 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:41:29.714394 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:29.714487 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:29.714638 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 16:41:29.714686 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:41:29.715883 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:29.710022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.3:36203)
0619 16:41:29.711110 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:29.711179 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:29.712517 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:29.713061 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:29.713069 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:29.713168 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:29.714071 (+ 903us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:41:29.714073 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:29.714117 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:41:29.714191 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:29.714478 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:29.714486 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:29.714494 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 16:41:29.714692 (+ 198us) client_negotiation.cc:240] Negotiation successful
0619 16:41:29.714959 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":198,"threads_started":1}
I20260619 16:41:29.717469 5466 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42343
I20260619 16:41:29.717810 5466 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:29.718416 5466 heartbeater.cc:511] Master 127.0.98.62:42343 requested a full tablet report, sending...
I20260619 16:41:29.719851 5016 ts_manager.cc:194] Registered new tserver with Master: 7000338d8b5a48589b2c427e9b9290a5 (127.0.98.3:43347)
I20260619 16:41:29.723315 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:41:29.727727 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:29.725679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51162 (local address 127.0.98.62:42343)
0619 16:41:29.725873 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:29.725879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:29.726025 (+ 146us) server_negotiation.cc:407] Connection header received
0619 16:41:29.726235 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:29.726244 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:29.726288 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:29.726376 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:29.727226 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:29.727230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:29.727233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:41:29.727253 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:29.727276 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:41:29.727285 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:29.727391 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:29.727455 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 16:41:29.727513 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 16:41:29.731422 5016 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:41:29.735517 5016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:41:29.751489 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:29.747247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43347 (local address 127.0.0.1:47460)
0619 16:41:29.748775 (+ 1528us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:29.748786 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:29.748897 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:29.749888 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:29.749893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:29.749914 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:29.750546 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:41:29.750549 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:29.750562 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:41:29.750583 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:29.751038 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:29.751047 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:29.751054 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 16:41:29.751169 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 16:41:29.751259 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1472,"thread_start_us":121,"threads_started":1}
I20260619 16:41:29.751534 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:29.747050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35695 (local address 127.0.0.1:57232)
0619 16:41:29.748345 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:29.748416 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:29.748542 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:29.749648 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:29.749652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:29.749672 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:29.750921 (+ 1249us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:41:29.750923 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:29.750939 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:41:29.750977 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:29.751320 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:29.751323 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:29.751326 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:41:29.751407 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 16:41:29.751436 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1064,"thread_start_us":155,"threads_started":1}
I20260619 16:41:29.751968 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:29.747078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44071 (local address 127.0.0.1:55596)
0619 16:41:29.748655 (+ 1577us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:29.748673 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:29.748795 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:29.749376 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:29.749384 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:29.749491 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:29.750262 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:41:29.750264 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:29.750294 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:41:29.750356 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:29.751611 (+ 1255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:29.751614 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:29.751620 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:41:29.751758 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 16:41:29.751805 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1469,"thread_start_us":139,"threads_started":1}
I20260619 16:41:29.752045 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:29.747624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47460 (local address 127.0.98.3:43347)
0619 16:41:29.748086 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:29.748101 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:29.748897 (+ 796us) server_negotiation.cc:407] Connection header received
0619 16:41:29.749021 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:29.749027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:29.749239 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:29.749416 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:29.750736 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:29.750744 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:29.750761 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:41:29.750777 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:29.750838 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:41:29.750874 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:29.750993 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:29.751752 (+ 759us) server_negotiation.cc:299] Negotiation successful
0619 16:41:29.751851 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":113,"threads_started":1}
I20260619 16:41:29.752254 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:29.748168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57232 (local address 127.0.98.1:35695)
0619 16:41:29.749062 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:29.749068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:29.749090 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:41:29.749140 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:29.749145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:29.749306 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:29.750309 (+ 1003us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:29.751134 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:29.751140 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:29.751148 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:41:29.751161 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:29.751195 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:41:29.751220 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:29.751331 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:29.751971 (+ 640us) server_negotiation.cc:299] Negotiation successful
0619 16:41:29.752055 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":792,"thread_start_us":131,"threads_started":1}
I20260619 16:41:29.752432 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:29.748401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55596 (local address 127.0.98.2:44071)
0619 16:41:29.748855 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:29.748869 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:29.748897 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:41:29.748957 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:29.748963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:29.749192 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:29.749349 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:29.751012 (+ 1663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:29.751022 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:29.751037 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:41:29.751052 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:29.751103 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:41:29.751136 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:29.751567 (+ 431us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:29.752159 (+ 592us) server_negotiation.cc:299] Negotiation successful
0619 16:41:29.752247 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":148,"threads_started":1}
I20260619 16:41:29.755899 5400 tablet_service.cc:1467] Processing CreateTablet for tablet 5c2df6fa716f4f71ab8208c2e00161dd (DEFAULT_TABLE table=test-table [id=cf8bf3b799284092876a58eb84fa9d10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:29.755877 5138 tablet_service.cc:1467] Processing CreateTablet for tablet 5c2df6fa716f4f71ab8208c2e00161dd (DEFAULT_TABLE table=test-table [id=cf8bf3b799284092876a58eb84fa9d10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:29.755887 5269 tablet_service.cc:1467] Processing CreateTablet for tablet 5c2df6fa716f4f71ab8208c2e00161dd (DEFAULT_TABLE table=test-table [id=cf8bf3b799284092876a58eb84fa9d10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:29.757202 5400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c2df6fa716f4f71ab8208c2e00161dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:29.757233 5138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c2df6fa716f4f71ab8208c2e00161dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:29.757233 5269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c2df6fa716f4f71ab8208c2e00161dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:29.765671 5485 tablet_bootstrap.cc:492] T 5c2df6fa716f4f71ab8208c2e00161dd P 3f6411f2a9e44486a207358821ad3de7: Bootstrap starting.
I20260619 16:41:29.767017 5483 tablet_bootstrap.cc:492] T 5c2df6fa716f4f71ab8208c2e00161dd P 7bc89a43b3404eaa83341b68be1c75bf: Bootstrap starting.
I20260619 16:41:29.767352 5484 tablet_bootstrap.cc:492] T 5c2df6fa716f4f71ab8208c2e00161dd P 7000338d8b5a48589b2c427e9b9290a5: Bootstrap starting.
I20260619 16:41:29.768949 5485 tablet_bootstrap.cc:654] T 5c2df6fa716f4f71ab8208c2e00161dd P 3f6411f2a9e44486a207358821ad3de7: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:29.769526 5483 tablet_bootstrap.cc:654] T 5c2df6fa716f4f71ab8208c2e00161dd P 7bc89a43b3404eaa83341b68be1c75bf: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:29.769727 5484 tablet_bootstrap.cc:654] T 5c2df6fa716f4f71ab8208c2e00161dd P 7000338d8b5a48589b2c427e9b9290a5: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:29.770388 5485 log.cc:824] T 5c2df6fa716f4f71ab8208c2e00161dd P 3f6411f2a9e44486a207358821ad3de7: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:29.770503 5484 log.cc:824] T 5c2df6fa716f4f71ab8208c2e00161dd P 7000338d8b5a48589b2c427e9b9290a5: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:29.770519 5483 log.cc:824] T 5c2df6fa716f4f71ab8208c2e00161dd P 7bc89a43b3404eaa83341b68be1c75bf: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:29.772908 5485 tablet_bootstrap.cc:492] T 5c2df6fa716f4f71ab8208c2e00161dd P 3f6411f2a9e44486a207358821ad3de7: No bootstrap required, opened a new log
I20260619 16:41:29.773021 5484 tablet_bootstrap.cc:492] T 5c2df6fa716f4f71ab8208c2e00161dd P 7000338d8b5a48589b2c427e9b9290a5: No bootstrap required, opened a new log
I20260619 16:41:29.773020 5483 tablet_bootstrap.cc:492] T 5c2df6fa716f4f71ab8208c2e00161dd P 7bc89a43b3404eaa83341b68be1c75bf: No bootstrap required, opened a new log
I20260619 16:41:29.773375 5484 ts_tablet_manager.cc:1397] T 5c2df6fa716f4f71ab8208c2e00161dd P 7000338d8b5a48589b2c427e9b9290a5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 16:41:29.773370 5485 ts_tablet_manager.cc:1397] T 5c2df6fa716f4f71ab8208c2e00161dd P 3f6411f2a9e44486a207358821ad3de7: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 16:41:29.773375 5483 ts_tablet_manager.cc:1397] T 5c2df6fa716f4f71ab8208c2e00161dd P 7bc89a43b3404eaa83341b68be1c75bf: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 16:41:29.778029 5485 raft_consensus.cc:348] T 5c2df6fa716f4f71ab8208c2e00161dd P 3f6411f2a9e44486a207358821ad3de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc89a43b3404eaa83341b68be1c75bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35695 } } peers { permanent_uuid: "3f6411f2a9e44486a207358821ad3de7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44071 } } peers { permanent_uuid: "7000338d8b5a48589b2c427e9b9290a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43347 } }
I20260619 16:41:29.778447 5485 raft_consensus.cc:374] T 5c2df6fa716f4f71ab8208c2e00161dd P 3f6411f2a9e44486a207358821ad3de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:29.778504 5485 raft_consensus.cc:729] T 5c2df6fa716f4f71ab8208c2e00161dd P 3f6411f2a9e44486a207358821ad3de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f6411f2a9e44486a207358821ad3de7, State: Initialized, Role: FOLLOWER
I20260619 16:41:29.779040 5485 consensus_queue.cc:260] T 5c2df6fa716f4f71ab8208c2e00161dd P 3f6411f2a9e44486a207358821ad3de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc89a43b3404eaa83341b68be1c75bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35695 } } peers { permanent_uuid: "3f6411f2a9e44486a207358821ad3de7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44071 } } peers { permanent_uuid: "7000338d8b5a48589b2c427e9b9290a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43347 } }
I20260619 16:41:29.778968 5483 raft_consensus.cc:348] T 5c2df6fa716f4f71ab8208c2e00161dd P 7bc89a43b3404eaa83341b68be1c75bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc89a43b3404eaa83341b68be1c75bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35695 } } peers { permanent_uuid: "3f6411f2a9e44486a207358821ad3de7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44071 } } peers { permanent_uuid: "7000338d8b5a48589b2c427e9b9290a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43347 } }
I20260619 16:41:29.779240 5483 raft_consensus.cc:374] T 5c2df6fa716f4f71ab8208c2e00161dd P 7bc89a43b3404eaa83341b68be1c75bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:29.779295 5483 raft_consensus.cc:729] T 5c2df6fa716f4f71ab8208c2e00161dd P 7bc89a43b3404eaa83341b68be1c75bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bc89a43b3404eaa83341b68be1c75bf, State: Initialized, Role: FOLLOWER
I20260619 16:41:29.779763 5483 consensus_queue.cc:260] T 5c2df6fa716f4f71ab8208c2e00161dd P 7bc89a43b3404eaa83341b68be1c75bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc89a43b3404eaa83341b68be1c75bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35695 } } peers { permanent_uuid: "3f6411f2a9e44486a207358821ad3de7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44071 } } peers { permanent_uuid: "7000338d8b5a48589b2c427e9b9290a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43347 } }
I20260619 16:41:29.779883 5335 heartbeater.cc:503] Master 127.0.98.62:42343 was elected leader, sending a full tablet report...
I20260619 16:41:29.779942 5484 raft_consensus.cc:348] T 5c2df6fa716f4f71ab8208c2e00161dd P 7000338d8b5a48589b2c427e9b9290a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc89a43b3404eaa83341b68be1c75bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35695 } } peers { permanent_uuid: "3f6411f2a9e44486a207358821ad3de7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44071 } } peers { permanent_uuid: "7000338d8b5a48589b2c427e9b9290a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43347 } }
I20260619 16:41:29.780184 5484 raft_consensus.cc:374] T 5c2df6fa716f4f71ab8208c2e00161dd P 7000338d8b5a48589b2c427e9b9290a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:29.780239 5484 raft_consensus.cc:729] T 5c2df6fa716f4f71ab8208c2e00161dd P 7000338d8b5a48589b2c427e9b9290a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7000338d8b5a48589b2c427e9b9290a5, State: Initialized, Role: FOLLOWER
I20260619 16:41:29.780314 5485 ts_tablet_manager.cc:1428] T 5c2df6fa716f4f71ab8208c2e00161dd P 3f6411f2a9e44486a207358821ad3de7: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260619 16:41:29.780673 5483 ts_tablet_manager.cc:1428] T 5c2df6fa716f4f71ab8208c2e00161dd P 7bc89a43b3404eaa83341b68be1c75bf: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260619 16:41:29.780717 5484 consensus_queue.cc:260] T 5c2df6fa716f4f71ab8208c2e00161dd P 7000338d8b5a48589b2c427e9b9290a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc89a43b3404eaa83341b68be1c75bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35695 } } peers { permanent_uuid: "3f6411f2a9e44486a207358821ad3de7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44071 } } peers { permanent_uuid: "7000338d8b5a48589b2c427e9b9290a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43347 } }
I20260619 16:41:29.781031 5204 heartbeater.cc:503] Master 127.0.98.62:42343 was elected leader, sending a full tablet report...
I20260619 16:41:29.781394 5466 heartbeater.cc:503] Master 127.0.98.62:42343 was elected leader, sending a full tablet report...
I20260619 16:41:29.781941 5484 ts_tablet_manager.cc:1428] T 5c2df6fa716f4f71ab8208c2e00161dd P 7000338d8b5a48589b2c427e9b9290a5: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
W20260619 16:41:29.957979 5467 tablet.cc:2369] T 5c2df6fa716f4f71ab8208c2e00161dd P 7000338d8b5a48589b2c427e9b9290a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:41:29.970112 5336 tablet.cc:2369] T 5c2df6fa716f4f71ab8208c2e00161dd P 3f6411f2a9e44486a207358821ad3de7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:41:29.989486 5205 tablet.cc:2369] T 5c2df6fa716f4f71ab8208c2e00161dd P 7bc89a43b3404eaa83341b68be1c75bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:41:30.137390 5490 raft_consensus.cc:482] T 5c2df6fa716f4f71ab8208c2e00161dd P 7bc89a43b3404eaa83341b68be1c75bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:41:30.137712 5490 raft_consensus.cc:504] T 5c2df6fa716f4f71ab8208c2e00161dd P 7bc89a43b3404eaa83341b68be1c75bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc89a43b3404eaa83341b68be1c75bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35695 } } peers { permanent_uuid: "3f6411f2a9e44486a207358821ad3de7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44071 } } peers { permanent_uuid: "7000338d8b5a48589b2c427e9b9290a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43347 } }
I20260619 16:41:30.139456 5490 leader_election.cc:288] T 5c2df6fa716f4f71ab8208c2e00161dd P 7bc89a43b3404eaa83341b68be1c75bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f6411f2a9e44486a207358821ad3de7 (127.0.98.2:44071), 7000338d8b5a48589b2c427e9b9290a5 (127.0.98.3:43347)
I20260619 16:41:30.141963 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:30.139919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56473 (local address 127.0.98.2:44071)
0619 16:41:30.140041 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:30.140047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:30.140348 (+ 301us) server_negotiation.cc:407] Connection header received
0619 16:41:30.140503 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:30.140507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:30.140549 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:30.140630 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:30.141495 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:30.141500 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:30.141505 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:41:30.141519 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:30.141541 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:41:30.141561 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:30.141628 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:30.141776 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 16:41:30.141818 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 16:41:30.142067 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:30.139848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51511 (local address 127.0.98.3:43347)
0619 16:41:30.140024 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:30.140029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:30.140329 (+ 300us) server_negotiation.cc:407] Connection header received
0619 16:41:30.140489 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:30.140493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:30.140545 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:30.140831 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:30.141662 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:30.141665 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:30.141668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:41:30.141703 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:30.141721 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:41:30.141730 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:30.141832 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:30.141907 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 16:41:30.141948 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:41:30.142082 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:30.139763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43347 (local address 127.0.98.1:51511)
0619 16:41:30.140219 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:30.140233 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:30.140374 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:30.140930 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:30.140937 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:30.140958 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:30.141491 (+ 533us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:41:30.141496 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:30.141507 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:41:30.141540 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:30.141826 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:30.141829 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:30.141832 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:41:30.141911 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 16:41:30.141945 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":159,"threads_started":1}
I20260619 16:41:30.142343 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:30.139763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44071 (local address 127.0.98.1:56473)
0619 16:41:30.140248 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:30.140261 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:30.140392 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:30.140644 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:30.140647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:30.140661 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:30.141294 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:41:30.141297 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:30.141311 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:41:30.141347 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:30.141684 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:30.141686 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:30.141688 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 16:41:30.141778 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 16:41:30.141818 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":200,"spinlock_wait_cycles":9984,"thread_start_us":111,"threads_started":1}
I20260619 16:41:30.142799 5420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c2df6fa716f4f71ab8208c2e00161dd" candidate_uuid: "7bc89a43b3404eaa83341b68be1c75bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7000338d8b5a48589b2c427e9b9290a5" is_pre_election: true
I20260619 16:41:30.142918 5289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c2df6fa716f4f71ab8208c2e00161dd" candidate_uuid: "7bc89a43b3404eaa83341b68be1c75bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f6411f2a9e44486a207358821ad3de7" is_pre_election: true
I20260619 16:41:30.143168 5420 raft_consensus.cc:2454] T 5c2df6fa716f4f71ab8208c2e00161dd P 7000338d8b5a48589b2c427e9b9290a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bc89a43b3404eaa83341b68be1c75bf in term 0.
I20260619 16:41:30.143225 5289 raft_consensus.cc:2454] T 5c2df6fa716f4f71ab8208c2e00161dd P 3f6411f2a9e44486a207358821ad3de7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bc89a43b3404eaa83341b68be1c75bf in term 0.
I20260619 16:41:30.143712 5092 leader_election.cc:302] T 5c2df6fa716f4f71ab8208c2e00161dd P 7bc89a43b3404eaa83341b68be1c75bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7000338d8b5a48589b2c427e9b9290a5, 7bc89a43b3404eaa83341b68be1c75bf; no voters:
I20260619 16:41:30.144028 5490 raft_consensus.cc:2781] T 5c2df6fa716f4f71ab8208c2e00161dd P 7bc89a43b3404eaa83341b68be1c75bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:41:30.144114 5490 raft_consensus.cc:482] T 5c2df6fa716f4f71ab8208c2e00161dd P 7bc89a43b3404eaa83341b68be1c75bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:41:30.144156 5490 raft_consensus.cc:3037] T 5c2df6fa716f4f71ab8208c2e00161dd P 7bc89a43b3404eaa83341b68be1c75bf [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:30.145197 5490 raft_consensus.cc:504] T 5c2df6fa716f4f71ab8208c2e00161dd P 7bc89a43b3404eaa83341b68be1c75bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc89a43b3404eaa83341b68be1c75bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35695 } } peers { permanent_uuid: "3f6411f2a9e44486a207358821ad3de7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44071 } } peers { permanent_uuid: "7000338d8b5a48589b2c427e9b9290a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43347 } }
I20260619 16:41:30.145550 5490 leader_election.cc:288] T 5c2df6fa716f4f71ab8208c2e00161dd P 7bc89a43b3404eaa83341b68be1c75bf [CANDIDATE]: Term 1 election: Requested vote from peers 3f6411f2a9e44486a207358821ad3de7 (127.0.98.2:44071), 7000338d8b5a48589b2c427e9b9290a5 (127.0.98.3:43347)
I20260619 16:41:30.145933 5420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c2df6fa716f4f71ab8208c2e00161dd" candidate_uuid: "7bc89a43b3404eaa83341b68be1c75bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7000338d8b5a48589b2c427e9b9290a5"
I20260619 16:41:30.145967 5289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c2df6fa716f4f71ab8208c2e00161dd" candidate_uuid: "7bc89a43b3404eaa83341b68be1c75bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f6411f2a9e44486a207358821ad3de7"
I20260619 16:41:30.146056 5420 raft_consensus.cc:3037] T 5c2df6fa716f4f71ab8208c2e00161dd P 7000338d8b5a48589b2c427e9b9290a5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:30.146116 5289 raft_consensus.cc:3037] T 5c2df6fa716f4f71ab8208c2e00161dd P 3f6411f2a9e44486a207358821ad3de7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:30.147150 5289 raft_consensus.cc:2454] T 5c2df6fa716f4f71ab8208c2e00161dd P 3f6411f2a9e44486a207358821ad3de7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bc89a43b3404eaa83341b68be1c75bf in term 1.
I20260619 16:41:30.147197 5420 raft_consensus.cc:2454] T 5c2df6fa716f4f71ab8208c2e00161dd P 7000338d8b5a48589b2c427e9b9290a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bc89a43b3404eaa83341b68be1c75bf in term 1.
I20260619 16:41:30.147495 5094 leader_election.cc:302] T 5c2df6fa716f4f71ab8208c2e00161dd P 7bc89a43b3404eaa83341b68be1c75bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f6411f2a9e44486a207358821ad3de7, 7bc89a43b3404eaa83341b68be1c75bf; no voters:
I20260619 16:41:30.147717 5490 raft_consensus.cc:2781] T 5c2df6fa716f4f71ab8208c2e00161dd P 7bc89a43b3404eaa83341b68be1c75bf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:30.147946 5490 raft_consensus.cc:686] T 5c2df6fa716f4f71ab8208c2e00161dd P 7bc89a43b3404eaa83341b68be1c75bf [term 1 LEADER]: Becoming Leader. State: Replica: 7bc89a43b3404eaa83341b68be1c75bf, State: Running, Role: LEADER
I20260619 16:41:30.148379 5490 consensus_queue.cc:237] T 5c2df6fa716f4f71ab8208c2e00161dd P 7bc89a43b3404eaa83341b68be1c75bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc89a43b3404eaa83341b68be1c75bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35695 } } peers { permanent_uuid: "3f6411f2a9e44486a207358821ad3de7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44071 } } peers { permanent_uuid: "7000338d8b5a48589b2c427e9b9290a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43347 } }
I20260619 16:41:30.151619 5013 catalog_manager.cc:5495] T 5c2df6fa716f4f71ab8208c2e00161dd P 7bc89a43b3404eaa83341b68be1c75bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bc89a43b3404eaa83341b68be1c75bf (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7bc89a43b3404eaa83341b68be1c75bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc89a43b3404eaa83341b68be1c75bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f6411f2a9e44486a207358821ad3de7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7000338d8b5a48589b2c427e9b9290a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43347 } health_report { overall_health: UNKNOWN } } }
I20260619 16:41:30.213951 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:30.212377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57240 (local address 127.0.98.1:35695)
0619 16:41:30.212506 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:30.212510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:30.212524 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:41:30.212649 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:30.212651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:30.212711 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:30.212781 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:30.213487 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:30.213490 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:30.213493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:41:30.213503 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:30.213521 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:41:30.213529 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:30.213611 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:30.213706 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 16:41:30.213776 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:41:30.220781 5289 raft_consensus.cc:1261] T 5c2df6fa716f4f71ab8208c2e00161dd P 3f6411f2a9e44486a207358821ad3de7 [term 1 FOLLOWER]: Refusing update from remote peer 7bc89a43b3404eaa83341b68be1c75bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:30.220777 5420 raft_consensus.cc:1261] T 5c2df6fa716f4f71ab8208c2e00161dd P 7000338d8b5a48589b2c427e9b9290a5 [term 1 FOLLOWER]: Refusing update from remote peer 7bc89a43b3404eaa83341b68be1c75bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:30.221848 5490 consensus_queue.cc:1035] T 5c2df6fa716f4f71ab8208c2e00161dd P 7bc89a43b3404eaa83341b68be1c75bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f6411f2a9e44486a207358821ad3de7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:30.222100 5494 consensus_queue.cc:1035] T 5c2df6fa716f4f71ab8208c2e00161dd P 7bc89a43b3404eaa83341b68be1c75bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7000338d8b5a48589b2c427e9b9290a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:30.231527 5498 mvcc.cc:205] Tried to move back new op lower bound from 7298610340733849600 to 7298610340449394688. Current Snapshot: MvccSnapshot[applied={T|T < 7298610340733849600}]
I20260619 16:41:30.242233 5015 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:41:30.242421 5015 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:41:30.245316 5015 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6778d1f8056741e289bf327ea6d1e8d0: Sending DeleteTablet for 3 replicas of tablet 5c2df6fa716f4f71ab8208c2e00161dd
I20260619 16:41:30.246156 5138 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c2df6fa716f4f71ab8208c2e00161dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:30 UTC) from {username='slave'} at 127.0.0.1:57232
I20260619 16:41:30.246153 5269 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c2df6fa716f4f71ab8208c2e00161dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:30 UTC) from {username='slave'} at 127.0.0.1:55596
I20260619 16:41:30.246845 5508 tablet_replica.cc:330] T 5c2df6fa716f4f71ab8208c2e00161dd P 7bc89a43b3404eaa83341b68be1c75bf: stopping tablet replica
I20260619 16:41:30.247102 5507 tablet_replica.cc:330] T 5c2df6fa716f4f71ab8208c2e00161dd P 3f6411f2a9e44486a207358821ad3de7: stopping tablet replica
I20260619 16:41:30.247311 5400 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c2df6fa716f4f71ab8208c2e00161dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:30 UTC) from {username='slave'} at 127.0.0.1:47460
I20260619 16:41:30.247763 5508 raft_consensus.cc:2229] T 5c2df6fa716f4f71ab8208c2e00161dd P 7bc89a43b3404eaa83341b68be1c75bf [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:41:30.247972 5507 raft_consensus.cc:2229] T 5c2df6fa716f4f71ab8208c2e00161dd P 3f6411f2a9e44486a207358821ad3de7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:41:30.248168 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 5076
I20260619 16:41:30.250043 5507 raft_consensus.cc:2258] T 5c2df6fa716f4f71ab8208c2e00161dd P 3f6411f2a9e44486a207358821ad3de7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:30.250481 5509 tablet_replica.cc:330] T 5c2df6fa716f4f71ab8208c2e00161dd P 7000338d8b5a48589b2c427e9b9290a5: stopping tablet replica
I20260619 16:41:30.250844 5509 raft_consensus.cc:2229] T 5c2df6fa716f4f71ab8208c2e00161dd P 7000338d8b5a48589b2c427e9b9290a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:41:30.251168 5509 raft_consensus.cc:2258] T 5c2df6fa716f4f71ab8208c2e00161dd P 7000338d8b5a48589b2c427e9b9290a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:30.251833 5507 ts_tablet_manager.cc:1905] T 5c2df6fa716f4f71ab8208c2e00161dd P 3f6411f2a9e44486a207358821ad3de7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:30.252635 5509 ts_tablet_manager.cc:1905] T 5c2df6fa716f4f71ab8208c2e00161dd P 7000338d8b5a48589b2c427e9b9290a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:30.255165 5507 ts_tablet_manager.cc:1918] T 5c2df6fa716f4f71ab8208c2e00161dd P 3f6411f2a9e44486a207358821ad3de7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:41:30.255288 5507 log.cc:1196] T 5c2df6fa716f4f71ab8208c2e00161dd P 3f6411f2a9e44486a207358821ad3de7: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-1/wal/wals/5c2df6fa716f4f71ab8208c2e00161dd
I20260619 16:41:30.255596 5509 ts_tablet_manager.cc:1918] T 5c2df6fa716f4f71ab8208c2e00161dd P 7000338d8b5a48589b2c427e9b9290a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:41:30.255707 5507 ts_tablet_manager.cc:1939] T 5c2df6fa716f4f71ab8208c2e00161dd P 3f6411f2a9e44486a207358821ad3de7: Deleting consensus metadata
I20260619 16:41:30.255709 5509 log.cc:1196] T 5c2df6fa716f4f71ab8208c2e00161dd P 7000338d8b5a48589b2c427e9b9290a5: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887259574980-392-0/minicluster-data/ts-2/wal/wals/5c2df6fa716f4f71ab8208c2e00161dd
I20260619 16:41:30.255704 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 5207
W20260619 16:41:30.255714 5002 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35695 (error 108)
I20260619 16:41:30.256982 5509 ts_tablet_manager.cc:1939] T 5c2df6fa716f4f71ab8208c2e00161dd P 7000338d8b5a48589b2c427e9b9290a5: Deleting consensus metadata
I20260619 16:41:30.258275 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:30.257835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35695 (local address 127.0.0.1:57248)
0619 16:41:30.258075 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:30.258163 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":69,"threads_started":1}
I20260619 16:41:30.258602 5001 catalog_manager.cc:4841] TS 7000338d8b5a48589b2c427e9b9290a5 (127.0.98.3:43347): tablet 5c2df6fa716f4f71ab8208c2e00161dd (table test-table [id=cf8bf3b799284092876a58eb84fa9d10]) successfully deleted
W20260619 16:41:30.258657 5002 catalog_manager.cc:4584] TS 7bc89a43b3404eaa83341b68be1c75bf (127.0.98.1:35695): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5c2df6fa716f4f71ab8208c2e00161dd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35695: connect: Connection refused (error 111)
I20260619 16:41:30.262490 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 5338
I20260619 16:41:30.263055 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:30.262715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44071 (local address 127.0.0.1:55604)
0619 16:41:30.262901 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:30.262953 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260619 16:41:30.268131 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 4985
2026-06-19T16:41:30Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1537 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 16:41:30.291087 392 mini_oidc.cc:152] Starting JWKS server
I20260619 16:41:30.292835 392 webserver.cc:460] Webserver started at https://127.0.0.1:43963/ using document root <none> and password file <none>
I20260619 16:41:30.293047 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 16:41:30.293349 392 webserver.cc:460] Webserver started at http://127.0.0.1:37953/ using document root <none> and password file <none>
2026-06-19T16:41:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:41:30Z Disabled control of system clock
I20260619 16:41:30.303357 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46675
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46675
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43963/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887259574980-392-0/testchainca.cert with env {}
W20260619 16:41:30.425513 5517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:30.425845 5517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:30.425890 5517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:30.430323 5517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:41:30.430405 5517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:30.430424 5517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:30.430445 5517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:41:30.430464 5517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 16:41:30.430528 5517 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 16:41:30.430552 5517 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43963/jwks.json
I20260619 16:41:30.436765 5517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46675
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43963/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887259574980-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:30.438002 5517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:30.439100 5517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:30.448652 5522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:30.448652 5523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:30.448882 5517 server_base.cc:1034] running on GCE node
W20260619 16:41:30.449015 5525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:30.449640 5517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:30.450881 5517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:30.452078 5517 hybrid_clock.cc:648] HybridClock initialized: now 1781887290452052 us; error 34 us; skew 500 ppm
I20260619 16:41:30.454058 5517 webserver.cc:460] Webserver started at http://127.0.98.62:43057/ using document root <none> and password file <none>
I20260619 16:41:30.454634 5517 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:30.454697 5517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:30.454877 5517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:30.457062 5517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "1de7370279204474a67094c27c679014"
format_stamp: "Formatted at 2026-06-19 16:41:30 on dist-test-slave-03cd"
server_key: "6e6541bd9d4e979d1ee4c15882a60482"
server_key_iv: "0b72747dd26dd2d370f4f9dccb7c86e6"
server_key_version: "encryptionkey@0"
I20260619 16:41:30.457590 5517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "1de7370279204474a67094c27c679014"
format_stamp: "Formatted at 2026-06-19 16:41:30 on dist-test-slave-03cd"
server_key: "6e6541bd9d4e979d1ee4c15882a60482"
server_key_iv: "0b72747dd26dd2d370f4f9dccb7c86e6"
server_key_version: "encryptionkey@0"
I20260619 16:41:30.461422 5517 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:41:30.464073 5531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:30.465513 5517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:41:30.465624 5517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "1de7370279204474a67094c27c679014"
format_stamp: "Formatted at 2026-06-19 16:41:30 on dist-test-slave-03cd"
server_key: "6e6541bd9d4e979d1ee4c15882a60482"
server_key_iv: "0b72747dd26dd2d370f4f9dccb7c86e6"
server_key_version: "encryptionkey@0"
I20260619 16:41:30.465723 5517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:30.502044 5517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:30.502744 5517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:30.502909 5517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:30.510223 5517 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46675
I20260619 16:41:30.510257 5583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46675 every 8 connection(s)
I20260619 16:41:30.511265 5517 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:41:30.514616 5584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:30.520241 5584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1de7370279204474a67094c27c679014: Bootstrap starting.
I20260619 16:41:30.520973 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 5517
I20260619 16:41:30.521091 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:41:30.521375 392 external_mini_cluster.cc:1428] Setting key 444f6b97b764bdb734ceeb72a88c2ea8
I20260619 16:41:30.523267 5584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1de7370279204474a67094c27c679014: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:30.524114 5584 log.cc:824] T 00000000000000000000000000000000 P 1de7370279204474a67094c27c679014: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:30.526386 5584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1de7370279204474a67094c27c679014: No bootstrap required, opened a new log
I20260619 16:41:30.530890 5584 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1de7370279204474a67094c27c679014 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de7370279204474a67094c27c679014" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } }
I20260619 16:41:30.531127 5584 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1de7370279204474a67094c27c679014 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:30.531162 5584 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1de7370279204474a67094c27c679014 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1de7370279204474a67094c27c679014, State: Initialized, Role: FOLLOWER
I20260619 16:41:30.531596 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:30.522778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52456 (local address 127.0.98.62:46675)
0619 16:41:30.523483 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:30.523498 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:30.523532 (+ 34us) server_negotiation.cc:407] Connection header received
0619 16:41:30.524310 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:30.524330 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:30.524929 (+ 599us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:30.525282 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:30.525673 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:30.526709 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:30.527939 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:30.528469 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:30.528597 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:30.529485 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:30.529490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:30.529503 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:41:30.529931 (+ 428us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:30.529989 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:41:30.530010 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:30.530087 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:30.530346 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 16:41:30.530702 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":168,"threads_started":1}
I20260619 16:41:30.531651 5584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1de7370279204474a67094c27c679014 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de7370279204474a67094c27c679014" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } }
I20260619 16:41:30.531867 5584 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1de7370279204474a67094c27c679014 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:41:30.531940 5584 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1de7370279204474a67094c27c679014 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:41:30.532078 5584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1de7370279204474a67094c27c679014 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:30.533200 5584 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1de7370279204474a67094c27c679014 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de7370279204474a67094c27c679014" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } }
I20260619 16:41:30.533562 5584 leader_election.cc:302] T 00000000000000000000000000000000 P 1de7370279204474a67094c27c679014 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1de7370279204474a67094c27c679014; no voters:
I20260619 16:41:30.533910 5584 leader_election.cc:288] T 00000000000000000000000000000000 P 1de7370279204474a67094c27c679014 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:41:30.534034 5589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1de7370279204474a67094c27c679014 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:30.534271 5589 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1de7370279204474a67094c27c679014 [term 1 LEADER]: Becoming Leader. State: Replica: 1de7370279204474a67094c27c679014, State: Running, Role: LEADER
I20260619 16:41:30.534600 5589 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1de7370279204474a67094c27c679014 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de7370279204474a67094c27c679014" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } }
I20260619 16:41:30.534915 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:41:30.535228 5584 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1de7370279204474a67094c27c679014 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:41:30.536197 5590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1de7370279204474a67094c27c679014 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1de7370279204474a67094c27c679014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de7370279204474a67094c27c679014" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } } }
I20260619 16:41:30.536319 5590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1de7370279204474a67094c27c679014 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:30.536834 5591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1de7370279204474a67094c27c679014 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1de7370279204474a67094c27c679014. Latest consensus state: current_term: 1 leader_uuid: "1de7370279204474a67094c27c679014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de7370279204474a67094c27c679014" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } } }
I20260619 16:41:30.536940 5591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1de7370279204474a67094c27c679014 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:30.537348 5598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:41:30.540807 5598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:41:30.547109 5598 catalog_manager.cc:1347] Generated new cluster ID: 69d4e0b306d24e74bd8a73f4ff8f61d1
I20260619 16:41:30.547224 5598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:41:30.568534 5598 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:41:30.569679 5598 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:41:30.578711 5598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1de7370279204474a67094c27c679014: Generated new TSK 0
I20260619 16:41:30.579571 5598 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 16:41:30.608978 5614 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 16:41:30.609148 5614 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 16:41:30.609782 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:30.591153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52462 (local address 127.0.98.62:46675)
0619 16:41:30.591295 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:30.591300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:30.591441 (+ 141us) server_negotiation.cc:407] Connection header received
0619 16:41:30.591642 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:30.591645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:30.591729 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:30.591849 (+ 120us) server_negotiation.cc:226] Negotiated authn=JWT
0619 16:41:30.592105 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:30.592960 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:30.593845 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:30.594195 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:30.594298 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:30.594654 (+ 356us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 16:41:30.609315 (+ 14661us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43963/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 16:41:30.609634 (+ 319us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52462: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43963/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 16:41:30.609870 5587 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52462: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43963/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 16:41:30.611076 5613 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:30.591021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46675 (local address 127.0.0.1:52462)
0619 16:41:30.591344 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:30.591358 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:30.591469 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:30.591858 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:30.591861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:30.591867 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0619 16:41:30.591989 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:30.591994 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:30.593090 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:30.593093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:30.593729 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:30.593737 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:30.594305 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:30.594325 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:30.594388 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:30.594439 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 16:41:30.609610 (+ 15171us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43963/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 16:41:30.609887 (+ 277us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:46675: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43963/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":120,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:46675: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43963/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 16:41:30.611807 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 5517
2026-06-19T16:41:30Z chronyd exiting
I20260619 16:41:30.649209 392 test_util.cc:175] -----------------------------------------------
I20260619 16:41:30.649325 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887259574980-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (359 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 16:41:30.650478 392 mini_oidc.cc:152] Starting JWKS server
I20260619 16:41:30.651393 392 webserver.cc:460] Webserver started at https://127.0.0.1:40269/ using document root <none> and password file <none>
I20260619 16:41:30.651525 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 16:41:30.651909 392 webserver.cc:460] Webserver started at http://127.0.0.1:43993/ using document root <none> and password file <none>
2026-06-19T16:41:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:41:30Z Disabled control of system clock
I20260619 16:41:30.660945 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44927
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44927
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40269/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887259574980-392-0/testchainca.cert with env {}
W20260619 16:41:30.774876 5620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:30.775211 5620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:30.775255 5620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:30.779630 5620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:41:30.779776 5620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:30.779808 5620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:30.779839 5620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:41:30.779865 5620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 16:41:30.779901 5620 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 16:41:30.779927 5620 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40269/jwks.json
I20260619 16:41:30.787184 5620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44927
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40269/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887259574980-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:30.788493 5620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:30.789592 5620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:30.799544 5626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:30.799618 5620 server_base.cc:1034] running on GCE node
W20260619 16:41:30.799515 5628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:30.799499 5625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:30.800565 5620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:30.801723 5620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:30.802912 5620 hybrid_clock.cc:648] HybridClock initialized: now 1781887290802897 us; error 28 us; skew 500 ppm
I20260619 16:41:30.805063 5620 webserver.cc:460] Webserver started at http://127.0.98.62:38955/ using document root <none> and password file <none>
I20260619 16:41:30.805749 5620 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:30.805833 5620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:30.806059 5620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:30.808329 5620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "49d0514eb3364f13ad8f66071d575010"
format_stamp: "Formatted at 2026-06-19 16:41:30 on dist-test-slave-03cd"
server_key: "65823527b3592da182331218da93c83b"
server_key_iv: "d5e548888be329f81fe8bbe6fdfd022a"
server_key_version: "encryptionkey@0"
I20260619 16:41:30.808933 5620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "49d0514eb3364f13ad8f66071d575010"
format_stamp: "Formatted at 2026-06-19 16:41:30 on dist-test-slave-03cd"
server_key: "65823527b3592da182331218da93c83b"
server_key_iv: "d5e548888be329f81fe8bbe6fdfd022a"
server_key_version: "encryptionkey@0"
I20260619 16:41:30.812995 5620 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:41:30.815527 5634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:30.816936 5620 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 16:41:30.817098 5620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "49d0514eb3364f13ad8f66071d575010"
format_stamp: "Formatted at 2026-06-19 16:41:30 on dist-test-slave-03cd"
server_key: "65823527b3592da182331218da93c83b"
server_key_iv: "d5e548888be329f81fe8bbe6fdfd022a"
server_key_version: "encryptionkey@0"
I20260619 16:41:30.817217 5620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:30.833443 5620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:30.834232 5620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:30.834465 5620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:30.841764 5620 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44927
I20260619 16:41:30.841805 5686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44927 every 8 connection(s)
I20260619 16:41:30.842808 5620 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:41:30.846344 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:30.848049 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 5620
I20260619 16:41:30.848198 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:41:30.848495 392 external_mini_cluster.cc:1428] Setting key 4fa81f0d9973078ba8193832f0b9e211
I20260619 16:41:30.853783 5687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49d0514eb3364f13ad8f66071d575010: Bootstrap starting.
I20260619 16:41:30.856671 5687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49d0514eb3364f13ad8f66071d575010: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:30.857631 5687 log.cc:824] T 00000000000000000000000000000000 P 49d0514eb3364f13ad8f66071d575010: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:30.860090 5690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:30.849969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33600 (local address 127.0.98.62:44927)
0619 16:41:30.850685 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:30.850708 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:30.850753 (+ 45us) server_negotiation.cc:407] Connection header received
0619 16:41:30.851708 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:30.851736 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:30.852306 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:30.852699 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:30.853210 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:30.854806 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:30.855879 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:30.856593 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:30.856765 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:30.857725 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:30.857734 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:30.857753 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:41:30.858277 (+ 524us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:30.858333 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:41:30.858354 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:30.858433 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:30.858696 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 16:41:30.858995 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":128,"threads_started":1}
I20260619 16:41:30.860281 5687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49d0514eb3364f13ad8f66071d575010: No bootstrap required, opened a new log
I20260619 16:41:30.863168 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:41:30.864589 5687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49d0514eb3364f13ad8f66071d575010 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d0514eb3364f13ad8f66071d575010" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44927 } }
I20260619 16:41:30.864895 5687 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49d0514eb3364f13ad8f66071d575010 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:30.864977 5687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49d0514eb3364f13ad8f66071d575010 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49d0514eb3364f13ad8f66071d575010, State: Initialized, Role: FOLLOWER
I20260619 16:41:30.865487 5687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49d0514eb3364f13ad8f66071d575010 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d0514eb3364f13ad8f66071d575010" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44927 } }
I20260619 16:41:30.865610 5687 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49d0514eb3364f13ad8f66071d575010 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:41:30.865677 5687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49d0514eb3364f13ad8f66071d575010 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:41:30.865777 5687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49d0514eb3364f13ad8f66071d575010 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:30.866891 5687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49d0514eb3364f13ad8f66071d575010 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d0514eb3364f13ad8f66071d575010" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44927 } }
I20260619 16:41:30.867379 5687 leader_election.cc:302] T 00000000000000000000000000000000 P 49d0514eb3364f13ad8f66071d575010 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49d0514eb3364f13ad8f66071d575010; no voters:
I20260619 16:41:30.867777 5687 leader_election.cc:288] T 00000000000000000000000000000000 P 49d0514eb3364f13ad8f66071d575010 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:41:30.867905 5692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49d0514eb3364f13ad8f66071d575010 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:30.868266 5692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49d0514eb3364f13ad8f66071d575010 [term 1 LEADER]: Becoming Leader. State: Replica: 49d0514eb3364f13ad8f66071d575010, State: Running, Role: LEADER
I20260619 16:41:30.868613 5692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49d0514eb3364f13ad8f66071d575010 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d0514eb3364f13ad8f66071d575010" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44927 } }
I20260619 16:41:30.869280 5687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49d0514eb3364f13ad8f66071d575010 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:41:30.870476 5693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49d0514eb3364f13ad8f66071d575010 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49d0514eb3364f13ad8f66071d575010" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d0514eb3364f13ad8f66071d575010" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44927 } } }
I20260619 16:41:30.870635 5693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49d0514eb3364f13ad8f66071d575010 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:30.870888 5694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49d0514eb3364f13ad8f66071d575010 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49d0514eb3364f13ad8f66071d575010. Latest consensus state: current_term: 1 leader_uuid: "49d0514eb3364f13ad8f66071d575010" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d0514eb3364f13ad8f66071d575010" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44927 } } }
I20260619 16:41:30.870983 5694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49d0514eb3364f13ad8f66071d575010 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:30.871121 5701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:41:30.874387 5701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:41:30.881150 5701 catalog_manager.cc:1347] Generated new cluster ID: 1b305383c9e64b7ba3fd2ff299f4a536
I20260619 16:41:30.881248 5701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:41:30.890378 5701 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:41:30.891780 5701 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:41:30.902401 5701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49d0514eb3364f13ad8f66071d575010: Generated new TSK 0
I20260619 16:41:30.903362 5701 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 16:41:30.930367 5717 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 16:41:30.930470 5717 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 16:41:30.931182 5690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:30.913367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33610 (local address 127.0.98.62:44927)
0619 16:41:30.913525 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:30.913532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:30.913734 (+ 202us) server_negotiation.cc:407] Connection header received
0619 16:41:30.913879 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:30.913881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:30.913944 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:30.914050 (+ 106us) server_negotiation.cc:226] Negotiated authn=JWT
0619 16:41:30.914568 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:30.915159 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:30.916534 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:30.916776 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:30.916875 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:30.917272 (+ 397us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 16:41:30.930795 (+ 13523us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40269/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 16:41:30.931045 (+ 250us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33610: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40269/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 16:41:30.931277 5690 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33610: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40269/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 16:41:30.931447 5716 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:30.913277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44927 (local address 127.0.0.1:33610)
0619 16:41:30.913604 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:30.913621 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:30.913752 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:30.914080 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:30.914083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:30.914092 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0619 16:41:30.914410 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:30.914419 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:30.915287 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:30.915291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:30.916325 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:30.916335 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:30.916895 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:30.916920 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:30.917020 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:30.917052 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 16:41:30.931032 (+ 13980us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40269/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 16:41:30.931260 (+ 228us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44927: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40269/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":115,"threads_started":1}
I20260619 16:41:30.932385 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 5620
2026-06-19T16:41:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (323 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 16:41:30.973843 392 mini_oidc.cc:152] Starting JWKS server
I20260619 16:41:30.974782 392 webserver.cc:460] Webserver started at https://127.0.0.1:41527/ using document root <none> and password file <none>
I20260619 16:41:30.974871 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 16:41:30.975297 392 webserver.cc:460] Webserver started at http://127.0.0.1:43813/ using document root <none> and password file <none>
2026-06-19T16:41:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:41:30Z Disabled control of system clock
I20260619 16:41:30.985404 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37125
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37125
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41527/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 16:41:31.107883 5723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:31.108248 5723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:31.108294 5723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:31.112495 5723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:41:31.112562 5723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:31.112579 5723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:31.112596 5723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:41:31.112612 5723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 16:41:31.112630 5723 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 16:41:31.112643 5723 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41527/jwks.json
I20260619 16:41:31.117710 5723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37125
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41527/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:31.119207 5723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:31.120687 5723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:31.129608 5729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:31.129722 5728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:31.129818 5723 server_base.cc:1034] running on GCE node
W20260619 16:41:31.129963 5731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:31.130587 5723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:31.132130 5723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:31.133337 5723 hybrid_clock.cc:648] HybridClock initialized: now 1781887291133307 us; error 46 us; skew 500 ppm
I20260619 16:41:31.135411 5723 webserver.cc:460] Webserver started at http://127.0.98.62:35313/ using document root <none> and password file <none>
I20260619 16:41:31.136107 5723 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:31.136209 5723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:31.136444 5723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:31.138629 5723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "025ffd4d6ab84933b4299e3b2b293bd9"
format_stamp: "Formatted at 2026-06-19 16:41:31 on dist-test-slave-03cd"
server_key: "502e6928f3073c47acdde86d195b85fb"
server_key_iv: "b3f17cad05d2fe3eb283a58f61ac3911"
server_key_version: "encryptionkey@0"
I20260619 16:41:31.139309 5723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "025ffd4d6ab84933b4299e3b2b293bd9"
format_stamp: "Formatted at 2026-06-19 16:41:31 on dist-test-slave-03cd"
server_key: "502e6928f3073c47acdde86d195b85fb"
server_key_iv: "b3f17cad05d2fe3eb283a58f61ac3911"
server_key_version: "encryptionkey@0"
I20260619 16:41:31.143327 5723 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 16:41:31.146351 5737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:31.147837 5723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 16:41:31.148036 5723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "025ffd4d6ab84933b4299e3b2b293bd9"
format_stamp: "Formatted at 2026-06-19 16:41:31 on dist-test-slave-03cd"
server_key: "502e6928f3073c47acdde86d195b85fb"
server_key_iv: "b3f17cad05d2fe3eb283a58f61ac3911"
server_key_version: "encryptionkey@0"
I20260619 16:41:31.148190 5723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:31.166702 5723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:31.167560 5723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:31.167850 5723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:31.175971 5723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37125
I20260619 16:41:31.175972 5789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37125 every 8 connection(s)
I20260619 16:41:31.177078 5723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:41:31.180718 5790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:31.182622 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 5723
I20260619 16:41:31.182746 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:41:31.183039 392 external_mini_cluster.cc:1428] Setting key 7a044302d92d166d86f7c2473371afd1
I20260619 16:41:31.188108 5790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025ffd4d6ab84933b4299e3b2b293bd9: Bootstrap starting.
I20260619 16:41:31.190604 5790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 025ffd4d6ab84933b4299e3b2b293bd9: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:31.191543 5790 log.cc:824] T 00000000000000000000000000000000 P 025ffd4d6ab84933b4299e3b2b293bd9: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:31.193257 5793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:31.184585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55668 (local address 127.0.98.62:37125)
0619 16:41:31.185355 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:31.185373 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:31.185410 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:41:31.186164 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:31.186184 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:31.186612 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:31.186981 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:31.187435 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:31.188516 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:31.189462 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:31.189955 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:31.190148 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:31.191266 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:31.191274 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:31.191288 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:41:31.191813 (+ 525us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:31.191917 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:41:31.191941 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:31.192056 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:31.192209 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 16:41:31.192489 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":147,"threads_started":1}
I20260619 16:41:31.194133 5790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025ffd4d6ab84933b4299e3b2b293bd9: No bootstrap required, opened a new log
I20260619 16:41:31.196659 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:41:31.198793 5790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 025ffd4d6ab84933b4299e3b2b293bd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025ffd4d6ab84933b4299e3b2b293bd9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37125 } }
I20260619 16:41:31.199056 5790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 025ffd4d6ab84933b4299e3b2b293bd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:31.199102 5790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 025ffd4d6ab84933b4299e3b2b293bd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 025ffd4d6ab84933b4299e3b2b293bd9, State: Initialized, Role: FOLLOWER
I20260619 16:41:31.199529 5790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 025ffd4d6ab84933b4299e3b2b293bd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025ffd4d6ab84933b4299e3b2b293bd9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37125 } }
I20260619 16:41:31.199730 5790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 025ffd4d6ab84933b4299e3b2b293bd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:41:31.199783 5790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 025ffd4d6ab84933b4299e3b2b293bd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:41:31.199853 5790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 025ffd4d6ab84933b4299e3b2b293bd9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:31.200830 5790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 025ffd4d6ab84933b4299e3b2b293bd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025ffd4d6ab84933b4299e3b2b293bd9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37125 } }
I20260619 16:41:31.201156 5790 leader_election.cc:302] T 00000000000000000000000000000000 P 025ffd4d6ab84933b4299e3b2b293bd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 025ffd4d6ab84933b4299e3b2b293bd9; no voters:
I20260619 16:41:31.201455 5790 leader_election.cc:288] T 00000000000000000000000000000000 P 025ffd4d6ab84933b4299e3b2b293bd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:41:31.201726 5795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 025ffd4d6ab84933b4299e3b2b293bd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:31.201992 5795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 025ffd4d6ab84933b4299e3b2b293bd9 [term 1 LEADER]: Becoming Leader. State: Replica: 025ffd4d6ab84933b4299e3b2b293bd9, State: Running, Role: LEADER
I20260619 16:41:31.202346 5795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 025ffd4d6ab84933b4299e3b2b293bd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025ffd4d6ab84933b4299e3b2b293bd9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37125 } }
I20260619 16:41:31.202920 5790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 025ffd4d6ab84933b4299e3b2b293bd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:41:31.204180 5797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025ffd4d6ab84933b4299e3b2b293bd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 025ffd4d6ab84933b4299e3b2b293bd9. Latest consensus state: current_term: 1 leader_uuid: "025ffd4d6ab84933b4299e3b2b293bd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025ffd4d6ab84933b4299e3b2b293bd9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37125 } } }
I20260619 16:41:31.204334 5797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025ffd4d6ab84933b4299e3b2b293bd9 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:31.204542 5796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025ffd4d6ab84933b4299e3b2b293bd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "025ffd4d6ab84933b4299e3b2b293bd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025ffd4d6ab84933b4299e3b2b293bd9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37125 } } }
I20260619 16:41:31.204636 5796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025ffd4d6ab84933b4299e3b2b293bd9 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:31.204900 5804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:41:31.208403 5804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:41:31.214934 5804 catalog_manager.cc:1347] Generated new cluster ID: 704e5af38fc141a2ab45f9ee4857a5dd
I20260619 16:41:31.215040 5804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:41:31.222833 5804 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:41:31.224171 5804 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:41:31.246680 5804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 025ffd4d6ab84933b4299e3b2b293bd9: Generated new TSK 0
I20260619 16:41:31.247488 5804 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:41:31.270025 5793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:31.262138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55684 (local address 127.0.98.62:37125)
0619 16:41:31.262296 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:31.262304 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:31.264027 (+ 1723us) server_negotiation.cc:407] Connection header received
0619 16:41:31.264286 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:31.264290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:31.264354 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:31.264452 (+ 98us) server_negotiation.cc:226] Negotiated authn=JWT
0619 16:41:31.264891 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:31.265785 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:31.267103 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:31.267469 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:31.267546 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:31.268074 (+ 528us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 16:41:31.269641 (+ 1567us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41527/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 16:41:31.269866 (+ 225us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55684: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41527/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 16:41:31.270099 5793 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55684: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41527/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 16:41:31.270679 5819 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:31.262350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37125 (local address 127.0.0.1:55684)
0619 16:41:31.263925 (+ 1575us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:31.263942 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:31.264088 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:31.264607 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:31.264610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:31.264618 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 16:41:31.264746 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:31.264755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:31.265941 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:31.265945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:31.266959 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:31.266969 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:31.267687 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:31.267718 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:31.267810 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:31.267843 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 16:41:31.270342 (+ 2499us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41527/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 16:41:31.270508 (+ 166us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37125: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41527/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":1486,"thread_start_us":126,"threads_started":1}
I20260619 16:41:31.274029 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 5723
E20260619 16:41:31.286357 5820 webserver.cc:533] Webserver: sslize failed:
E20260619 16:41:31.286480 5820 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T16:41:31Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (341 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T16:41:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:41:31Z Disabled control of system clock
I20260619 16:41:31.324218 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34623
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34623
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887259574980-392-0/testchainca.cert with env {}
W20260619 16:41:31.444677 5824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:31.445118 5824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:31.445199 5824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:31.449513 5824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:41:31.449597 5824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:31.449613 5824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:31.449631 5824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:41:31.449647 5824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 16:41:31.449664 5824 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 16:41:31.449678 5824 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 16:41:31.455511 5824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34623
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887259574980-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:31.456661 5824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:31.457659 5824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:31.466732 5830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:31.466749 5832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:31.466749 5829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:31.467007 5824 server_base.cc:1034] running on GCE node
I20260619 16:41:31.467737 5824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:31.469015 5824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:31.470232 5824 hybrid_clock.cc:648] HybridClock initialized: now 1781887291470205 us; error 37 us; skew 500 ppm
I20260619 16:41:31.472430 5824 webserver.cc:460] Webserver started at http://127.0.98.62:42621/ using document root <none> and password file <none>
I20260619 16:41:31.473011 5824 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:31.473076 5824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:31.473245 5824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:31.475409 5824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "53ff94a61db94188845430fb8ee38dd8"
format_stamp: "Formatted at 2026-06-19 16:41:31 on dist-test-slave-03cd"
server_key: "4531870eb27a8f10d631247e37e5e726"
server_key_iv: "1614e5955850d3ff968e3b9ac13d4305"
server_key_version: "encryptionkey@0"
I20260619 16:41:31.476083 5824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "53ff94a61db94188845430fb8ee38dd8"
format_stamp: "Formatted at 2026-06-19 16:41:31 on dist-test-slave-03cd"
server_key: "4531870eb27a8f10d631247e37e5e726"
server_key_iv: "1614e5955850d3ff968e3b9ac13d4305"
server_key_version: "encryptionkey@0"
I20260619 16:41:31.480324 5824 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:41:31.483213 5838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:31.485000 5824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:41:31.485173 5824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "53ff94a61db94188845430fb8ee38dd8"
format_stamp: "Formatted at 2026-06-19 16:41:31 on dist-test-slave-03cd"
server_key: "4531870eb27a8f10d631247e37e5e726"
server_key_iv: "1614e5955850d3ff968e3b9ac13d4305"
server_key_version: "encryptionkey@0"
I20260619 16:41:31.485325 5824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:31.501776 5824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:31.502542 5824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:31.502770 5824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:31.510897 5824 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34623
I20260619 16:41:31.510862 5890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34623 every 8 connection(s)
I20260619 16:41:31.511965 5824 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:41:31.515259 5891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:31.521234 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 5824
I20260619 16:41:31.521347 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:41:31.521569 5891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53ff94a61db94188845430fb8ee38dd8: Bootstrap starting.
I20260619 16:41:31.521643 392 external_mini_cluster.cc:1428] Setting key 6f1bad249850a53afc1b0e541dcfcd0c
I20260619 16:41:31.524314 5891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53ff94a61db94188845430fb8ee38dd8: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:31.525074 5891 log.cc:824] T 00000000000000000000000000000000 P 53ff94a61db94188845430fb8ee38dd8: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:31.527381 5891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53ff94a61db94188845430fb8ee38dd8: No bootstrap required, opened a new log
I20260619 16:41:31.531407 5894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:31.523021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43134 (local address 127.0.98.62:34623)
0619 16:41:31.523789 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:31.523808 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:31.523844 (+ 36us) server_negotiation.cc:407] Connection header received
0619 16:41:31.524577 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:31.524595 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:31.525010 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:31.525341 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:31.525814 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:31.526824 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:31.528042 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:31.528521 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:31.528644 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:31.529539 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:31.529544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:31.529557 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:41:31.529967 (+ 410us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:31.530024 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:41:31.530044 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:31.530121 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:31.530292 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 16:41:31.530618 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":145,"threads_started":1}
I20260619 16:41:31.531806 5891 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53ff94a61db94188845430fb8ee38dd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ff94a61db94188845430fb8ee38dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34623 } }
I20260619 16:41:31.532060 5891 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53ff94a61db94188845430fb8ee38dd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:31.532133 5891 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53ff94a61db94188845430fb8ee38dd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53ff94a61db94188845430fb8ee38dd8, State: Initialized, Role: FOLLOWER
I20260619 16:41:31.532591 5891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53ff94a61db94188845430fb8ee38dd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ff94a61db94188845430fb8ee38dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34623 } }
I20260619 16:41:31.532753 5891 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53ff94a61db94188845430fb8ee38dd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:41:31.532846 5891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53ff94a61db94188845430fb8ee38dd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:41:31.532971 5891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53ff94a61db94188845430fb8ee38dd8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:31.534111 5891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53ff94a61db94188845430fb8ee38dd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ff94a61db94188845430fb8ee38dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34623 } }
I20260619 16:41:31.534466 5891 leader_election.cc:302] T 00000000000000000000000000000000 P 53ff94a61db94188845430fb8ee38dd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53ff94a61db94188845430fb8ee38dd8; no voters:
I20260619 16:41:31.534727 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:41:31.534866 5891 leader_election.cc:288] T 00000000000000000000000000000000 P 53ff94a61db94188845430fb8ee38dd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:41:31.534986 5896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53ff94a61db94188845430fb8ee38dd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:31.535292 5896 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53ff94a61db94188845430fb8ee38dd8 [term 1 LEADER]: Becoming Leader. State: Replica: 53ff94a61db94188845430fb8ee38dd8, State: Running, Role: LEADER
I20260619 16:41:31.535660 5896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53ff94a61db94188845430fb8ee38dd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ff94a61db94188845430fb8ee38dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34623 } }
I20260619 16:41:31.536262 5891 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53ff94a61db94188845430fb8ee38dd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:41:31.537497 5897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53ff94a61db94188845430fb8ee38dd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53ff94a61db94188845430fb8ee38dd8. Latest consensus state: current_term: 1 leader_uuid: "53ff94a61db94188845430fb8ee38dd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ff94a61db94188845430fb8ee38dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34623 } } }
I20260619 16:41:31.537657 5897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53ff94a61db94188845430fb8ee38dd8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:31.538185 5898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53ff94a61db94188845430fb8ee38dd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53ff94a61db94188845430fb8ee38dd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ff94a61db94188845430fb8ee38dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34623 } } }
I20260619 16:41:31.538316 5898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53ff94a61db94188845430fb8ee38dd8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:31.538206 5906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:41:31.542255 5906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:41:31.548493 5906 catalog_manager.cc:1347] Generated new cluster ID: 609580e2eb70423f952a4b7a4d84b1a6
I20260619 16:41:31.548588 5906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:41:31.586068 5906 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:41:31.587370 5906 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:41:31.598587 5906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53ff94a61db94188845430fb8ee38dd8: Generated new TSK 0
I20260619 16:41:31.599581 5906 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:41:31.653599 5894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:31.614904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43140 (local address 127.0.98.62:34623)
0619 16:41:31.615048 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:31.615054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:31.615222 (+ 168us) server_negotiation.cc:407] Connection header received
0619 16:41:31.615372 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:31.615375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:31.615438 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:31.615563 (+ 125us) server_negotiation.cc:226] Negotiated authn=JWT
0619 16:41:31.615839 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:31.616693 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:31.617621 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:31.617960 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:31.618068 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:31.618372 (+ 304us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 16:41:31.653077 (+ 34705us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 16:41:31.653425 (+ 348us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43140: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 16:41:31.653699 5894 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43140: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 16:41:31.653826 5920 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:31.614770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34623 (local address 127.0.0.1:43140)
0619 16:41:31.615139 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:31.615152 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:31.615245 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:31.615560 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:31.615562 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:31.615568 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0619 16:41:31.615721 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:31.615732 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:31.616846 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:31.616850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:31.617503 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:31.617511 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:31.618061 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:31.618084 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:31.618150 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:31.618175 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 16:41:31.653373 (+ 35198us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 16:41:31.653582 (+ 209us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34623: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":107,"threads_started":1}
I20260619 16:41:31.654621 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 5824
2026-06-19T16:41:31Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (358 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (131 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (133 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:41:31 dist-test-slave-03cd krb5kdc[5932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:41:31 dist-test-slave-03cd krb5kdc[5932](info): set up 2 sockets
Jun 19 16:41:31 dist-test-slave-03cd krb5kdc[5932](info): commencing operation
krb5kdc: starting...
W20260619 16:41:33.020877 392 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:41:33 dist-test-slave-03cd krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:41:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:41:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:41:33.163398 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37331
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:33.302134 5945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:33.302578 5945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:33.302680 5945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:33.307770 5945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:41:33.307886 5945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:33.307921 5945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:33.307965 5945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:41:33.308024 5945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:41:33.314851 5945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:33.316797 5945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:33.318240 5945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:33.328884 5953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:33.328890 5951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:33.328946 5950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:33.329345 5945 server_base.cc:1034] running on GCE node
I20260619 16:41:33.329998 5945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:33.331348 5945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:33.332540 5945 hybrid_clock.cc:648] HybridClock initialized: now 1781887293332512 us; error 61 us; skew 500 ppm
Jun 19 16:41:33 dist-test-slave-03cd krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:33.342275 5945 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:41:33.343564 5945 webserver.cc:460] Webserver started at http://127.0.98.62:37817/ using document root <none> and password file <none>
I20260619 16:41:33.344352 5945 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:33.344425 5945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:33.344640 5945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:33.347221 5945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "25c4fdb15d59448fadecc21840db1664"
format_stamp: "Formatted at 2026-06-19 16:41:33 on dist-test-slave-03cd"
server_key: "c763f0fb0e75e1bbba67fed857118c8c"
server_key_iv: "7ee3ff1351e9d76362f74a200b3e6225"
server_key_version: "encryptionkey@0"
I20260619 16:41:33.348150 5945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "25c4fdb15d59448fadecc21840db1664"
format_stamp: "Formatted at 2026-06-19 16:41:33 on dist-test-slave-03cd"
server_key: "c763f0fb0e75e1bbba67fed857118c8c"
server_key_iv: "7ee3ff1351e9d76362f74a200b3e6225"
server_key_version: "encryptionkey@0"
I20260619 16:41:33.353615 5945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:41:33.356897 5960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:33.358752 5945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:41:33.358934 5945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "25c4fdb15d59448fadecc21840db1664"
format_stamp: "Formatted at 2026-06-19 16:41:33 on dist-test-slave-03cd"
server_key: "c763f0fb0e75e1bbba67fed857118c8c"
server_key_iv: "7ee3ff1351e9d76362f74a200b3e6225"
server_key_version: "encryptionkey@0"
I20260619 16:41:33.359174 5945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:33.384135 5945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:33.388896 5945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:33.389186 5945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:33.399130 5945 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37331
I20260619 16:41:33.399528 6012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37331 every 8 connection(s)
I20260619 16:41:33.400550 5945 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:41:33.402398 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 5945
I20260619 16:41:33.402570 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:41:33.402897 392 external_mini_cluster.cc:1428] Setting key ed49dad1245fcb91904dd4f27d3ba6a6
I20260619 16:41:33.404734 6013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:33.412104 6013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25c4fdb15d59448fadecc21840db1664: Bootstrap starting.
Jun 19 16:41:33 dist-test-slave-03cd krb5kdc[5932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:33.414633 6013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25c4fdb15d59448fadecc21840db1664: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:33.415848 6013 log.cc:824] T 00000000000000000000000000000000 P 25c4fdb15d59448fadecc21840db1664: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:33.418120 6013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25c4fdb15d59448fadecc21840db1664: No bootstrap required, opened a new log
I20260619 16:41:33.420873 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:33.404658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33350 (local address 127.0.98.62:37331)
0619 16:41:33.405236 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:33.405257 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:33.405306 (+ 49us) server_negotiation.cc:407] Connection header received
0619 16:41:33.406240 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:33.406262 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:33.406709 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:33.407095 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:33.408049 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:33.409568 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:33.410580 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:33.411303 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:33.411492 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:33.414515 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:33.414524 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:33.414547 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:33.414594 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:33.417153 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:33.417764 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:33.417771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:33.417777 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:33.417919 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:33.418243 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:33.418251 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:33.418252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:33.418816 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:33.419057 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:33.419379 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 16:41:33.419825 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":178,"threads_started":1}
I20260619 16:41:33.422451 6013 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25c4fdb15d59448fadecc21840db1664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c4fdb15d59448fadecc21840db1664" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37331 } }
I20260619 16:41:33.422780 6013 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25c4fdb15d59448fadecc21840db1664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:33.422873 6013 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25c4fdb15d59448fadecc21840db1664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25c4fdb15d59448fadecc21840db1664, State: Initialized, Role: FOLLOWER
I20260619 16:41:33.423417 6013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25c4fdb15d59448fadecc21840db1664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c4fdb15d59448fadecc21840db1664" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37331 } }
I20260619 16:41:33.423615 6013 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25c4fdb15d59448fadecc21840db1664 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:41:33.423786 6013 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25c4fdb15d59448fadecc21840db1664 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:41:33.423915 6013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25c4fdb15d59448fadecc21840db1664 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:33.425061 6013 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25c4fdb15d59448fadecc21840db1664 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c4fdb15d59448fadecc21840db1664" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37331 } }
I20260619 16:41:33.425467 6013 leader_election.cc:302] T 00000000000000000000000000000000 P 25c4fdb15d59448fadecc21840db1664 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25c4fdb15d59448fadecc21840db1664; no voters:
I20260619 16:41:33.425963 6013 leader_election.cc:288] T 00000000000000000000000000000000 P 25c4fdb15d59448fadecc21840db1664 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:41:33.426126 6018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25c4fdb15d59448fadecc21840db1664 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:33.426487 6018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25c4fdb15d59448fadecc21840db1664 [term 1 LEADER]: Becoming Leader. State: Replica: 25c4fdb15d59448fadecc21840db1664, State: Running, Role: LEADER
I20260619 16:41:33.426944 6018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25c4fdb15d59448fadecc21840db1664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c4fdb15d59448fadecc21840db1664" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37331 } }
I20260619 16:41:33.427446 6013 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25c4fdb15d59448fadecc21840db1664 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:41:33.429209 6020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25c4fdb15d59448fadecc21840db1664 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25c4fdb15d59448fadecc21840db1664. Latest consensus state: current_term: 1 leader_uuid: "25c4fdb15d59448fadecc21840db1664" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c4fdb15d59448fadecc21840db1664" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37331 } } }
I20260619 16:41:33.429386 6020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25c4fdb15d59448fadecc21840db1664 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:33.429745 6019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25c4fdb15d59448fadecc21840db1664 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25c4fdb15d59448fadecc21840db1664" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c4fdb15d59448fadecc21840db1664" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37331 } } }
I20260619 16:41:33.429841 6019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25c4fdb15d59448fadecc21840db1664 [sys.catalog]: This master's current role is: LEADER
W20260619 16:41:33.433158 6033 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 25c4fdb15d59448fadecc21840db1664: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:41:33.433297 6033 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:41:33.433470 6031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:41:33.434648 6031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:41:33.442256 6031 catalog_manager.cc:1347] Generated new cluster ID: 0406386903ac43ea891f68f85ee8d340
I20260619 16:41:33.442384 6031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:41:33.478776 6031 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:41:33.480301 6031 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:41:33.487437 6031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25c4fdb15d59448fadecc21840db1664: Generated new TSK 0
I20260619 16:41:33.488560 6031 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:41:33.532753 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37331
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:33.683183 6039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:33.683493 6039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:33.683538 6039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:33.688086 6039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:33.688186 6039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:33.688252 6039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:41:33.693960 6039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:33.695230 6039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:33.696403 6039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:33.707430 6045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:33.707440 6044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:33.707495 6047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:33.707958 6039 server_base.cc:1034] running on GCE node
I20260619 16:41:33.708459 6039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:33.709120 6039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:33.710314 6039 hybrid_clock.cc:648] HybridClock initialized: now 1781887293710272 us; error 79 us; skew 500 ppm
Jun 19 16:41:33 dist-test-slave-03cd krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:33.714287 6039 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:41:33.715386 6039 webserver.cc:460] Webserver started at http://127.0.98.1:44845/ using document root <none> and password file <none>
I20260619 16:41:33.716080 6039 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:33.716151 6039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:33.716379 6039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:33.718583 6039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-0/data/instance:
uuid: "21fa06b780d54137adc8e9920158cd7e"
format_stamp: "Formatted at 2026-06-19 16:41:33 on dist-test-slave-03cd"
server_key: "ca1c7564189a1b7e59fd7be1dec38836"
server_key_iv: "a6d85e33d0c1aa44df28cd863da571b1"
server_key_version: "encryptionkey@0"
I20260619 16:41:33.719185 6039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-0/wal/instance:
uuid: "21fa06b780d54137adc8e9920158cd7e"
format_stamp: "Formatted at 2026-06-19 16:41:33 on dist-test-slave-03cd"
server_key: "ca1c7564189a1b7e59fd7be1dec38836"
server_key_iv: "a6d85e33d0c1aa44df28cd863da571b1"
server_key_version: "encryptionkey@0"
I20260619 16:41:33.723842 6039 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 16:41:33.726907 6054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:33.728771 6039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:41:33.728950 6039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "21fa06b780d54137adc8e9920158cd7e"
format_stamp: "Formatted at 2026-06-19 16:41:33 on dist-test-slave-03cd"
server_key: "ca1c7564189a1b7e59fd7be1dec38836"
server_key_iv: "a6d85e33d0c1aa44df28cd863da571b1"
server_key_version: "encryptionkey@0"
I20260619 16:41:33.729115 6039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:33.762792 6039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:33.766716 6039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:33.766963 6039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:33.767629 6039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:33.768621 6039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:33.768699 6039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:33.768760 6039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:33.768812 6039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:33.779632 6039 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35223
I20260619 16:41:33.779697 6167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35223 every 8 connection(s)
I20260619 16:41:33.780645 6039 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:41:33.783564 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 6039
I20260619 16:41:33.783875 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:41:33.784238 392 external_mini_cluster.cc:1428] Setting key e0365f4e32b0315473d751cbf4e9a21c
Jun 19 16:41:33 dist-test-slave-03cd krb5kdc[5932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:41:33.799901 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:33.783013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45385 (local address 127.0.98.62:37331)
0619 16:41:33.783239 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:33.783243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:33.784166 (+ 923us) server_negotiation.cc:407] Connection header received
0619 16:41:33.785741 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:33.785744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:33.785807 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:33.785921 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:33.787543 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:33.788478 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:33.789782 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:33.790239 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:33.790386 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:33.794005 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:33.794011 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:33.794014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:33.794063 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:33.796573 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:33.797462 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:33.797465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:33.797467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:33.797559 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:33.798043 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:33.798061 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:33.798062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:33.798365 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:33.798458 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:33.799475 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0619 16:41:33.799643 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 16:41:33.801147 6170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:33.783410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37331 (local address 127.0.98.1:45385)
0619 16:41:33.783954 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:33.784008 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:33.785528 (+ 1520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:33.786121 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:33.786130 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:33.786629 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:33.787345 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:33.787367 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:33.788648 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:33.788651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:33.789634 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:33.789645 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:33.790419 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:33.790421 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:33.790588 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:33.791362 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:33.791379 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:33.793683 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:33.796791 (+ 3108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:33.796800 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:33.796825 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:33.797310 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:33.797716 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:33.797719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:33.797720 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:33.797929 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:33.798516 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:33.798528 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:33.799107 (+ 579us) client_negotiation.cc:769] Sending connection context
0619 16:41:33.799415 (+ 308us) client_negotiation.cc:240] Negotiation successful
0619 16:41:33.799990 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":166,"threads_started":1}
I20260619 16:41:33.802883 6168 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37331
I20260619 16:41:33.803258 6168 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:33.803954 6168 heartbeater.cc:511] Master 127.0.98.62:37331 requested a full tablet report, sending...
I20260619 16:41:33.806169 5977 ts_manager.cc:194] Registered new tserver with Master: 21fa06b780d54137adc8e9920158cd7e (127.0.98.1:35223)
I20260619 16:41:33.808192 5977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:41:33.819972 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37331
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:33.962195 6173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:33.962606 6173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:33.962659 6173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:33.967559 6173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:33.967686 6173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:33.967762 6173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:41:33.974220 6173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:33.975739 6173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:33.977043 6173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:41:33.990796 6173 server_base.cc:1034] running on GCE node
W20260619 16:41:33.990833 6178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:33.990833 6181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:33.991039 6179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:33.991711 6173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:33.992501 6173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:33.993707 6173 hybrid_clock.cc:648] HybridClock initialized: now 1781887293993663 us; error 67 us; skew 500 ppm
Jun 19 16:41:33 dist-test-slave-03cd krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:33.997347 6173 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:41:33.998495 6173 webserver.cc:460] Webserver started at http://127.0.98.2:36551/ using document root <none> and password file <none>
I20260619 16:41:33.999470 6173 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:33.999601 6173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:33.999933 6173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:34.002388 6173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-1/data/instance:
uuid: "9addfa3f1ce94532be30557a261ec708"
format_stamp: "Formatted at 2026-06-19 16:41:33 on dist-test-slave-03cd"
server_key: "f67076bf6b24c061ac1a507d7e3d9fa2"
server_key_iv: "d4e4c2f944dd189ea5e030daac524453"
server_key_version: "encryptionkey@0"
I20260619 16:41:34.003084 6173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9addfa3f1ce94532be30557a261ec708"
format_stamp: "Formatted at 2026-06-19 16:41:33 on dist-test-slave-03cd"
server_key: "f67076bf6b24c061ac1a507d7e3d9fa2"
server_key_iv: "d4e4c2f944dd189ea5e030daac524453"
server_key_version: "encryptionkey@0"
I20260619 16:41:34.008808 6173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 16:41:34.012188 6188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:34.013957 6173 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:41:34.014196 6173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "9addfa3f1ce94532be30557a261ec708"
format_stamp: "Formatted at 2026-06-19 16:41:33 on dist-test-slave-03cd"
server_key: "f67076bf6b24c061ac1a507d7e3d9fa2"
server_key_iv: "d4e4c2f944dd189ea5e030daac524453"
server_key_version: "encryptionkey@0"
I20260619 16:41:34.014353 6173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:34.056910 6173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:34.061738 6173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:34.062001 6173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:34.062698 6173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:34.064061 6173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:34.064143 6173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:34.064213 6173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:34.064251 6173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:34.075078 6173 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36311
I20260619 16:41:34.075152 6301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36311 every 8 connection(s)
I20260619 16:41:34.076299 6173 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:41:34.079888 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 6173
I20260619 16:41:34.080053 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:41:34.080422 392 external_mini_cluster.cc:1428] Setting key dc5a5c95410eea4b86307a575417b588
Jun 19 16:41:34 dist-test-slave-03cd krb5kdc[5932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 16:41:34.096514 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:34.078860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49771 (local address 127.0.98.62:37331)
0619 16:41:34.079079 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:34.079084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:34.080553 (+ 1469us) server_negotiation.cc:407] Connection header received
0619 16:41:34.082747 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:34.082751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:34.082817 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:34.082894 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:34.084859 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:34.085456 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:34.086947 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:34.087208 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:34.087352 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:34.091319 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:34.091327 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:34.091332 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:34.091386 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:34.093493 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:34.094320 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:34.094323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:34.094325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:34.094432 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:34.094873 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:34.094875 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:34.094877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:34.095175 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:34.095314 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:34.096189 (+ 875us) server_negotiation.cc:299] Negotiation successful
0619 16:41:34.096352 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 16:41:34.097834 6304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:34.079404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37331 (local address 127.0.98.2:49771)
0619 16:41:34.080267 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:34.080353 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:34.082394 (+ 2041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:34.083250 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:34.083262 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:34.083905 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:34.084612 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:34.084632 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:34.085633 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:34.085637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:34.086753 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:34.086779 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:34.087423 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:34.087428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:34.087758 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:34.088676 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:34.088709 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:34.091004 (+ 2295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:34.093715 (+ 2711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:34.093723 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:34.093741 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:34.094148 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:34.094557 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:34.094559 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:34.094561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:34.094753 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:34.095331 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:34.095337 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:34.095817 (+ 480us) client_negotiation.cc:769] Sending connection context
0619 16:41:34.096082 (+ 265us) client_negotiation.cc:240] Negotiation successful
0619 16:41:34.096535 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":225,"threads_started":1}
I20260619 16:41:34.099354 6302 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37331
I20260619 16:41:34.099792 6302 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:34.100658 6302 heartbeater.cc:511] Master 127.0.98.62:37331 requested a full tablet report, sending...
I20260619 16:41:34.102507 5977 ts_manager.cc:194] Registered new tserver with Master: 9addfa3f1ce94532be30557a261ec708 (127.0.98.2:36311)
I20260619 16:41:34.103209 5977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:41:34.116537 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37331
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:34.253252 6307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:34.253649 6307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:34.253702 6307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:34.258396 6307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:34.258504 6307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:34.258579 6307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:41:34.265123 6307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:34.266633 6307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:34.268050 6307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:34.280131 6315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:34.280140 6312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:34.280140 6313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:34.280652 6307 server_base.cc:1034] running on GCE node
I20260619 16:41:34.281206 6307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:34.282053 6307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:34.283272 6307 hybrid_clock.cc:648] HybridClock initialized: now 1781887294283235 us; error 83 us; skew 500 ppm
Jun 19 16:41:34 dist-test-slave-03cd krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:34.287397 6307 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:41:34.289110 6307 webserver.cc:460] Webserver started at http://127.0.98.3:46391/ using document root <none> and password file <none>
I20260619 16:41:34.290030 6307 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:34.290145 6307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:34.290444 6307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:34.293790 6307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-2/data/instance:
uuid: "c2953b3d2e884f8ba0f25355cc9a5055"
format_stamp: "Formatted at 2026-06-19 16:41:34 on dist-test-slave-03cd"
server_key: "e99f279d1ebf223a561300d86113e716"
server_key_iv: "45dd121f0215580ec676db2d0d4e9cc9"
server_key_version: "encryptionkey@0"
I20260619 16:41:34.294625 6307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c2953b3d2e884f8ba0f25355cc9a5055"
format_stamp: "Formatted at 2026-06-19 16:41:34 on dist-test-slave-03cd"
server_key: "e99f279d1ebf223a561300d86113e716"
server_key_iv: "45dd121f0215580ec676db2d0d4e9cc9"
server_key_version: "encryptionkey@0"
I20260619 16:41:34.300833 6307 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260619 16:41:34.305189 6322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:34.307219 6307 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260619 16:41:34.307462 6307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "c2953b3d2e884f8ba0f25355cc9a5055"
format_stamp: "Formatted at 2026-06-19 16:41:34 on dist-test-slave-03cd"
server_key: "e99f279d1ebf223a561300d86113e716"
server_key_iv: "45dd121f0215580ec676db2d0d4e9cc9"
server_key_version: "encryptionkey@0"
I20260619 16:41:34.307739 6307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:34.325133 6307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:34.330207 6307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:34.330493 6307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:34.331349 6307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:34.332571 6307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:34.332657 6307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:34.332724 6307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:34.332765 6307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:34.345901 6307 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35951
I20260619 16:41:34.346004 6435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35951 every 8 connection(s)
I20260619 16:41:34.347186 6307 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:41:34.356202 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 6307
I20260619 16:41:34.356382 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:41:34.356706 392 external_mini_cluster.cc:1428] Setting key c3b50db7349508107c392af24b39cd3c
Jun 19 16:41:34 dist-test-slave-03cd krb5kdc[5932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:34.371060 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:34.350720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37541 (local address 127.0.98.62:37331)
0619 16:41:34.350963 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:34.350969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:34.352619 (+ 1650us) server_negotiation.cc:407] Connection header received
0619 16:41:34.355529 (+ 2910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:34.355534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:34.355618 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:34.355868 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:34.358370 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:34.359235 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:34.360901 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:34.361164 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:34.361294 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:34.364984 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:34.364991 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:34.364996 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:34.365049 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:34.367225 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:34.368370 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:34.368374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:34.368375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:34.368501 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:34.369059 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:34.369061 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:34.369063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:34.369321 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:34.369437 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:34.370575 (+ 1138us) server_negotiation.cc:299] Negotiation successful
0619 16:41:34.370820 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 16:41:34.372179 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:34.351224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37331 (local address 127.0.98.3:37541)
0619 16:41:34.352342 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:34.352448 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:34.355048 (+ 2600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:34.356142 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:34.356156 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:34.357027 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:34.358032 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:34.358061 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:34.359456 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:34.359461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:34.360736 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:34.360752 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:34.361357 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:34.361361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:34.361595 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:34.362561 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:34.362582 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:34.364658 (+ 2076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:34.367463 (+ 2805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:34.367471 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:34.367530 (+ 59us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:34.368120 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:34.368645 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:34.368648 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:34.368650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:34.368880 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:34.369490 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:34.369498 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:34.369880 (+ 382us) client_negotiation.cc:769] Sending connection context
0619 16:41:34.370235 (+ 355us) client_negotiation.cc:240] Negotiation successful
0619 16:41:34.370722 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"thread_start_us":473,"threads_started":1}
I20260619 16:41:34.373648 6436 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37331
I20260619 16:41:34.374042 6436 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:34.374738 6436 heartbeater.cc:511] Master 127.0.98.62:37331 requested a full tablet report, sending...
I20260619 16:41:34.376132 5977 ts_manager.cc:194] Registered new tserver with Master: c2953b3d2e884f8ba0f25355cc9a5055 (127.0.98.3:35951)
I20260619 16:41:34.376894 5977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37541
I20260619 16:41:34.384861 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/test-admin.keytab.
Jun 19 16:41:34 dist-test-slave-03cd krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:34.404235 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 16:41:34.407761 6443 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:34.406004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57500 (local address 127.0.98.1:35223)
0619 16:41:34.406453 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:34.406462 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:34.406497 (+ 35us) server_negotiation.cc:407] Connection header received
0619 16:41:34.406567 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:34.406572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:34.406835 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:34.407006 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:34.407560 (+ 554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57500: BlockingRecv error: recv got EOF from 127.0.0.1:57500 (error 108)
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":151,"threads_started":1}
W20260619 16:41:34.407975 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 16:41:34 dist-test-slave-03cd krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:34.409409 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 16:41:34 dist-test-slave-03cd krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:34.410604 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 16:41:34 dist-test-slave-03cd krb5kdc[5932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:41:34.423574 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:34.411891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57514 (local address 127.0.98.1:35223)
0619 16:41:34.412101 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:34.412105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:34.412362 (+ 257us) server_negotiation.cc:407] Connection header received
0619 16:41:34.412476 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:34.412479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:34.412536 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:34.412630 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:34.413467 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:34.414293 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:34.415418 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:34.415921 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:34.416024 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:34.419030 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:34.419046 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:34.419066 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:34.419118 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:34.421285 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:34.421849 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:34.421855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:34.421857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:34.421965 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:34.422312 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:34.422315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:34.422317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:34.422702 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:34.422910 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:34.423202 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 16:41:34.423382 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
W20260619 16:41:34.429396 6443 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:34.427732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57516 (local address 127.0.98.1:35223)
0619 16:41:34.427969 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:34.427974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:34.428164 (+ 190us) server_negotiation.cc:407] Connection header received
0619 16:41:34.428432 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:34.428434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:34.428492 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:34.428574 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:34.429256 (+ 682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57516: BlockingRecv error: recv got EOF from 127.0.0.1:57516 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
W20260619 16:41:34.429401 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 16:41:34 dist-test-slave-03cd krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:34.430812 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 16:41:34 dist-test-slave-03cd krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:34.431995 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 16:41:34 dist-test-slave-03cd krb5kdc[5932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:41:34.443972 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:34.433395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57520 (local address 127.0.98.1:35223)
0619 16:41:34.433696 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:34.433701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:34.433840 (+ 139us) server_negotiation.cc:407] Connection header received
0619 16:41:34.434019 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:34.434022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:34.434098 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:34.434199 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:34.434825 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:34.435358 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:34.436256 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:34.436529 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:34.436653 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:34.440419 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:34.440423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:34.440426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:34.440458 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:34.442066 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:34.442666 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:34.442669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:34.442671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:34.442753 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:34.443043 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:34.443045 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:34.443047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:34.443225 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:34.443353 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:34.443629 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 16:41:34.443839 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192}
W20260619 16:41:34.447862 6443 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:34.446312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57536 (local address 127.0.98.1:35223)
0619 16:41:34.446504 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:34.446509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:34.446650 (+ 141us) server_negotiation.cc:407] Connection header received
0619 16:41:34.446834 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:34.446837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:34.446898 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:34.446994 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:34.447696 (+ 702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57536: BlockingRecv error: recv got EOF from 127.0.0.1:57536 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260619 16:41:34.448016 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887259574980-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 16:41:34 dist-test-slave-03cd krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:34.449455 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 16:41:34 dist-test-slave-03cd krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:34.450577 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 16:41:34 dist-test-slave-03cd krb5kdc[5932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:41:34.462491 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:34.451962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57540 (local address 127.0.98.1:35223)
0619 16:41:34.452127 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:34.452133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:34.452282 (+ 149us) server_negotiation.cc:407] Connection header received
0619 16:41:34.452473 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:34.452481 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:34.452530 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:34.452631 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:34.453216 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:34.453880 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:34.454730 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:34.454967 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:34.455080 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:34.458256 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:34.458260 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:34.458263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:34.458301 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:34.460235 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:34.460869 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:34.460872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:34.460874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:34.461053 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:34.461541 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:34.461544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:34.461545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:34.461805 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:34.461964 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:34.462199 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 16:41:34.462335 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:41:34.464159 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 6039
I20260619 16:41:34.472216 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 6173
I20260619 16:41:34.478911 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 6307
I20260619 16:41:34.485220 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 5945
2026-06-19T16:41:34Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2570 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:41:34 dist-test-slave-03cd krb5kdc[6456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:41:34 dist-test-slave-03cd krb5kdc[6456](info): set up 2 sockets
Jun 19 16:41:34 dist-test-slave-03cd krb5kdc[6456](info): commencing operation
krb5kdc: starting...
W20260619 16:41:35.560794 392 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:41:35 dist-test-slave-03cd krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:41:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:41:35Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260619 16:41:35.704463 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43729
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43729
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:35.842458 6469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:35.842888 6469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:35.842972 6469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:35.847718 6469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:41:35.847856 6469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:35.847892 6469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:35.847925 6469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:41:35.847978 6469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:41:35.854400 6469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:35.856251 6469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:35.857724 6469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:41:35.869251 6469 server_base.cc:1034] running on GCE node
W20260619 16:41:35.869130 6475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:35.869124 6477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:35.869130 6474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:35.870416 6469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:35.872092 6469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:35.873297 6469 hybrid_clock.cc:648] HybridClock initialized: now 1781887295873244 us; error 81 us; skew 500 ppm
Jun 19 16:41:35 dist-test-slave-03cd krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887295, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:35.876926 6469 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260619 16:41:35.878189 6469 webserver.cc:460] Webserver started at http://127.0.98.62:45997/ using document root <none> and password file <none>
I20260619 16:41:35.878902 6469 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:35.878974 6469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:35.879191 6469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:35.881958 6469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "d328d7aca0674111beef001a9009b0f8"
format_stamp: "Formatted at 2026-06-19 16:41:35 on dist-test-slave-03cd"
server_key: "0c32ddaddd8f028bcc559ea9ec773554"
server_key_iv: "23e359f4c60f7fbdef05da4cdb4943c1"
server_key_version: "encryptionkey@0"
I20260619 16:41:35.882627 6469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "d328d7aca0674111beef001a9009b0f8"
format_stamp: "Formatted at 2026-06-19 16:41:35 on dist-test-slave-03cd"
server_key: "0c32ddaddd8f028bcc559ea9ec773554"
server_key_iv: "23e359f4c60f7fbdef05da4cdb4943c1"
server_key_version: "encryptionkey@0"
I20260619 16:41:35.887790 6469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260619 16:41:35.891288 6484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:35.893441 6469 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:41:35.893641 6469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "d328d7aca0674111beef001a9009b0f8"
format_stamp: "Formatted at 2026-06-19 16:41:35 on dist-test-slave-03cd"
server_key: "0c32ddaddd8f028bcc559ea9ec773554"
server_key_iv: "23e359f4c60f7fbdef05da4cdb4943c1"
server_key_version: "encryptionkey@0"
I20260619 16:41:35.893795 6469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:35.920396 6469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:35.924911 6469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:35.925122 6469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:35.934883 6469 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43729
I20260619 16:41:35.935247 6536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43729 every 8 connection(s)
I20260619 16:41:35.936319 6469 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:41:35.939762 6537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:35.944628 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 6469
I20260619 16:41:35.944772 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:41:35.945079 392 external_mini_cluster.cc:1428] Setting key 2618f787f7a528a1e67fb483c65d1f7e
I20260619 16:41:35.946305 6537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8: Bootstrap starting.
I20260619 16:41:35.948982 6537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:35.949939 6537 log.cc:824] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:35.952837 6537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8: No bootstrap required, opened a new log
Jun 19 16:41:35 dist-test-slave-03cd krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:41:35.957830 6537 raft_consensus.cc:348] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } }
I20260619 16:41:35.958216 6537 raft_consensus.cc:374] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:35.958324 6537 raft_consensus.cc:729] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d328d7aca0674111beef001a9009b0f8, State: Initialized, Role: FOLLOWER
I20260619 16:41:35.958868 6537 consensus_queue.cc:260] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } }
I20260619 16:41:35.959039 6537 raft_consensus.cc:388] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:41:35.959134 6537 raft_consensus.cc:482] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:41:35.959268 6537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:35.960932 6537 raft_consensus.cc:504] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } }
I20260619 16:41:35.961409 6537 leader_election.cc:302] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d328d7aca0674111beef001a9009b0f8; no voters:
I20260619 16:41:35.961906 6537 leader_election.cc:288] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:41:35.962028 6542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:35.962251 6542 raft_consensus.cc:686] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [term 1 LEADER]: Becoming Leader. State: Replica: d328d7aca0674111beef001a9009b0f8, State: Running, Role: LEADER
I20260619 16:41:35.962612 6542 consensus_queue.cc:237] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } }
I20260619 16:41:35.963411 6537 sys_catalog.cc:564] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:41:35.964083 6542 sys_catalog.cc:455] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d328d7aca0674111beef001a9009b0f8. Latest consensus state: current_term: 1 leader_uuid: "d328d7aca0674111beef001a9009b0f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } }
I20260619 16:41:35.964224 6542 sys_catalog.cc:458] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:35.964695 6543 sys_catalog.cc:455] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d328d7aca0674111beef001a9009b0f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } }
I20260619 16:41:35.964910 6543 sys_catalog.cc:458] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:35.965332 6548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:41:35.965387 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:35.947016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40264 (local address 127.0.98.62:43729)
0619 16:41:35.947610 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:35.947630 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:35.947732 (+ 102us) server_negotiation.cc:407] Connection header received
0619 16:41:35.948608 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:35.948631 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:35.949127 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:35.949530 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:35.950543 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:35.951999 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:35.953302 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:35.953959 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:35.954130 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:35.957609 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:35.957617 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:35.957639 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:35.957684 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:35.960606 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:35.961357 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:35.961373 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:35.961381 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:35.961623 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:35.962006 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:35.962009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:35.962011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:35.962866 (+ 855us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:35.963116 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:35.963476 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 16:41:35.963944 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":212,"threads_started":1}
I20260619 16:41:35.969616 6548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:41:35.978392 6548 catalog_manager.cc:1347] Generated new cluster ID: 02c7ab1904fc44fdb1154a0c0059f3af
I20260619 16:41:35.978525 6548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:41:36.006161 6548 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:41:36.007701 6548 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:41:36.017266 6548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8: Generated new TSK 0
I20260619 16:41:36.018198 6548 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260619 16:41:36.065006 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43729
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:36.232673 6563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:36.233037 6563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:36.233116 6563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:36.237504 6563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:36.237627 6563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:36.237732 6563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:41:36.244315 6563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:36.245704 6563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:36.246929 6563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:36.258982 6569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:36.259037 6568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:36.259277 6571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:36.259397 6563 server_base.cc:1034] running on GCE node
I20260619 16:41:36.260098 6563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:36.260970 6563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:36.262190 6563 hybrid_clock.cc:648] HybridClock initialized: now 1781887296262168 us; error 43 us; skew 500 ppm
Jun 19 16:41:36 dist-test-slave-03cd krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887296, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:36.266145 6563 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 16:41:36.267321 6563 webserver.cc:460] Webserver started at http://127.0.98.1:42149/ using document root <none> and password file <none>
I20260619 16:41:36.268141 6563 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:36.268234 6563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:36.268504 6563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:36.271010 6563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/data/instance:
uuid: "715aa0a6a1334d59a2680f0b8cef1c85"
format_stamp: "Formatted at 2026-06-19 16:41:36 on dist-test-slave-03cd"
server_key: "11402e67b3772e9be1d01d3f9b963491"
server_key_iv: "4442af73e6a46c27b6d6d3d522244f67"
server_key_version: "encryptionkey@0"
I20260619 16:41:36.271744 6563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/wal/instance:
uuid: "715aa0a6a1334d59a2680f0b8cef1c85"
format_stamp: "Formatted at 2026-06-19 16:41:36 on dist-test-slave-03cd"
server_key: "11402e67b3772e9be1d01d3f9b963491"
server_key_iv: "4442af73e6a46c27b6d6d3d522244f67"
server_key_version: "encryptionkey@0"
I20260619 16:41:36.276644 6563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 16:41:36.279950 6578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:36.281701 6563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:41:36.281880 6563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "715aa0a6a1334d59a2680f0b8cef1c85"
format_stamp: "Formatted at 2026-06-19 16:41:36 on dist-test-slave-03cd"
server_key: "11402e67b3772e9be1d01d3f9b963491"
server_key_iv: "4442af73e6a46c27b6d6d3d522244f67"
server_key_version: "encryptionkey@0"
I20260619 16:41:36.282025 6563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:36.310286 6563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:36.314549 6563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:36.314802 6563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:36.315560 6563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:36.316612 6563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:36.316684 6563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:36.316759 6563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:36.316800 6563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:36.328926 6563 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41841
I20260619 16:41:36.328963 6691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41841 every 8 connection(s)
I20260619 16:41:36.330112 6563 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:41:36.335490 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 6563
I20260619 16:41:36.335721 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:41:36.336092 392 external_mini_cluster.cc:1428] Setting key 3b6a044d995d04b1cbfa3715b1bc1ebb
Jun 19 16:41:36 dist-test-slave-03cd krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887296, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:41:36.348929 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:36.332706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59183 (local address 127.0.98.62:43729)
0619 16:41:36.332926 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:36.332930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:36.334145 (+ 1215us) server_negotiation.cc:407] Connection header received
0619 16:41:36.335888 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:36.335894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:36.336001 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:36.336160 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:36.337848 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:36.338519 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:36.340065 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:36.340329 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:36.340452 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:36.344292 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:36.344295 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:36.344297 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:36.344328 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:36.346232 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:36.346959 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:36.346962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:36.346964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:36.347053 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:36.347454 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:36.347458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:36.347460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:36.347784 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:36.347896 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:36.348554 (+ 658us) server_negotiation.cc:299] Negotiation successful
0619 16:41:36.348721 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260619 16:41:36.350044 6694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:36.333217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.98.1:59183)
0619 16:41:36.333928 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:36.333992 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:36.335490 (+ 1498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:36.336431 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:36.336440 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:36.336981 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:36.337651 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:36.337671 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:36.338733 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:36.338737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:36.339921 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:36.339935 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:36.340519 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:36.340523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:36.340799 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:36.341669 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:36.341699 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:36.344041 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:36.346398 (+ 2357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:36.346407 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:36.346430 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:36.346776 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:36.347152 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:36.347157 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:36.347159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:36.347342 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:36.347895 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:36.347905 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:36.348252 (+ 347us) client_negotiation.cc:769] Sending connection context
0619 16:41:36.348505 (+ 253us) client_negotiation.cc:240] Negotiation successful
0619 16:41:36.348950 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":191,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260619 16:41:36.351941 6692 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43729
I20260619 16:41:36.352269 6692 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:36.353123 6692 heartbeater.cc:511] Master 127.0.98.62:43729 requested a full tablet report, sending...
I20260619 16:41:36.355055 6501 ts_manager.cc:194] Registered new tserver with Master: 715aa0a6a1334d59a2680f0b8cef1c85 (127.0.98.1:41841)
I20260619 16:41:36.356402 6501 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260619 16:41:36.372419 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43729
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:36.504050 6697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:36.504423 6697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:36.504514 6697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:36.509308 6697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:36.509447 6697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:36.509598 6697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:41:36.515872 6697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:36.517153 6697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:36.518318 6697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:36.529954 6705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:36.530002 6702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:36.529954 6703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:36.530547 6697 server_base.cc:1034] running on GCE node
I20260619 16:41:36.531136 6697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:36.532074 6697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:36.533290 6697 hybrid_clock.cc:648] HybridClock initialized: now 1781887296533263 us; error 44 us; skew 500 ppm
Jun 19 16:41:36 dist-test-slave-03cd krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887296, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:36.536845 6697 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 16:41:36.538058 6697 webserver.cc:460] Webserver started at http://127.0.98.2:40787/ using document root <none> and password file <none>
I20260619 16:41:36.538810 6697 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:36.538914 6697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:36.539204 6697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:36.541929 6697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/data/instance:
uuid: "da90af8c16574881832a55a5861d2c8f"
format_stamp: "Formatted at 2026-06-19 16:41:36 on dist-test-slave-03cd"
server_key: "bc7fdef778629ae2f9a299aaa4d24c65"
server_key_iv: "3d40d03c9cf986dcfac787c09963b357"
server_key_version: "encryptionkey@0"
I20260619 16:41:36.542651 6697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/wal/instance:
uuid: "da90af8c16574881832a55a5861d2c8f"
format_stamp: "Formatted at 2026-06-19 16:41:36 on dist-test-slave-03cd"
server_key: "bc7fdef778629ae2f9a299aaa4d24c65"
server_key_iv: "3d40d03c9cf986dcfac787c09963b357"
server_key_version: "encryptionkey@0"
I20260619 16:41:36.547633 6697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:41:36.551275 6712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:36.553171 6697 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260619 16:41:36.553406 6697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "da90af8c16574881832a55a5861d2c8f"
format_stamp: "Formatted at 2026-06-19 16:41:36 on dist-test-slave-03cd"
server_key: "bc7fdef778629ae2f9a299aaa4d24c65"
server_key_iv: "3d40d03c9cf986dcfac787c09963b357"
server_key_version: "encryptionkey@0"
I20260619 16:41:36.553608 6697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:36.583201 6697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:36.587636 6697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:36.587918 6697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:36.588744 6697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:36.589823 6697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:36.589905 6697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:36.589982 6697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:36.590034 6697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:36.602846 6697 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41495
I20260619 16:41:36.602949 6825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41495 every 8 connection(s)
I20260619 16:41:36.604137 6697 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:41:36.612115 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 6697
I20260619 16:41:36.612237 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:41:36.612602 392 external_mini_cluster.cc:1428] Setting key 9655f4dd5248b0c8d388b3808ef8664f
Jun 19 16:41:36 dist-test-slave-03cd krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887296, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:41:36.623454 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:36.606453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38751 (local address 127.0.98.62:43729)
0619 16:41:36.606627 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:36.606632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:36.607681 (+ 1049us) server_negotiation.cc:407] Connection header received
0619 16:41:36.609373 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:36.609377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:36.609449 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:36.609557 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:36.611081 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:36.611738 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:36.613249 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:36.613677 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:36.613807 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:36.617934 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:36.617942 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:36.617948 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:36.617997 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:36.620367 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:36.621157 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:36.621164 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:36.621168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:36.621294 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:36.621833 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:36.621840 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:36.621843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:36.622175 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:36.622303 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:36.623008 (+ 705us) server_negotiation.cc:299] Negotiation successful
0619 16:41:36.623197 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:41:36.624742 6828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:36.606875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.98.2:38751)
0619 16:41:36.607448 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:36.607504 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:36.609082 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:36.609788 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:36.609797 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:36.610280 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:36.610904 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:36.610920 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:36.611872 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:36.611878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:36.613044 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:36.613057 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:36.613893 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:36.613898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:36.614127 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:36.615094 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:36.615118 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:36.617610 (+ 2492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:36.620577 (+ 2967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:36.620585 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:36.620603 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:36.621010 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:36.621505 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:36.621508 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:36.621509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:36.621701 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:36.622324 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:36.622330 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:36.622699 (+ 369us) client_negotiation.cc:769] Sending connection context
0619 16:41:36.622905 (+ 206us) client_negotiation.cc:240] Negotiation successful
0619 16:41:36.623244 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":153,"threads_started":1}
I20260619 16:41:36.626925 6826 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43729
I20260619 16:41:36.627230 6826 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:36.628060 6826 heartbeater.cc:511] Master 127.0.98.62:43729 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260619 16:41:36.629410 6501 ts_manager.cc:194] Registered new tserver with Master: da90af8c16574881832a55a5861d2c8f (127.0.98.2:41495)
I20260619 16:41:36.630090 6501 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 16:41:36.649410 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43729
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:36.770838 6831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:36.771169 6831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:36.771215 6831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:36.775493 6831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:36.775593 6831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:36.775682 6831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:41:36.781445 6831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:36.782575 6831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:36.783607 6831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:36.795171 6836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:36.795133 6837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:36.795121 6839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:36.795941 6831 server_base.cc:1034] running on GCE node
I20260619 16:41:36.796460 6831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:36.797425 6831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:36.798766 6831 hybrid_clock.cc:648] HybridClock initialized: now 1781887296798697 us; error 93 us; skew 500 ppm
Jun 19 16:41:36 dist-test-slave-03cd krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887296, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:36.802608 6831 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 16:41:36.804307 6831 webserver.cc:460] Webserver started at http://127.0.98.3:45809/ using document root <none> and password file <none>
I20260619 16:41:36.805006 6831 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:36.805124 6831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:36.805392 6831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:36.808714 6831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/data/instance:
uuid: "a22a1178390044a9a7cc957008be869f"
format_stamp: "Formatted at 2026-06-19 16:41:36 on dist-test-slave-03cd"
server_key: "7bb284ee0b3b82fe57353e7345dc4054"
server_key_iv: "43fd3cbf89e68985338474ed7ab23ff3"
server_key_version: "encryptionkey@0"
I20260619 16:41:36.809705 6831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a22a1178390044a9a7cc957008be869f"
format_stamp: "Formatted at 2026-06-19 16:41:36 on dist-test-slave-03cd"
server_key: "7bb284ee0b3b82fe57353e7345dc4054"
server_key_iv: "43fd3cbf89e68985338474ed7ab23ff3"
server_key_version: "encryptionkey@0"
I20260619 16:41:36.815236 6831 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 16:41:36.818697 6846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:36.820866 6831 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 16:41:36.821066 6831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "a22a1178390044a9a7cc957008be869f"
format_stamp: "Formatted at 2026-06-19 16:41:36 on dist-test-slave-03cd"
server_key: "7bb284ee0b3b82fe57353e7345dc4054"
server_key_iv: "43fd3cbf89e68985338474ed7ab23ff3"
server_key_version: "encryptionkey@0"
I20260619 16:41:36.821238 6831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:36.842247 6831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:36.846283 6831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:36.846553 6831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:36.847245 6831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:36.848766 6831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:36.849098 6831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:36.849188 6831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:36.849236 6831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:36.861631 6959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34813 every 8 connection(s)
I20260619 16:41:36.861649 6831 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34813
I20260619 16:41:36.862911 6831 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:41:36.868700 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 6831
I20260619 16:41:36.868872 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:41:36.869164 392 external_mini_cluster.cc:1428] Setting key 5198aec42111a8d47d1f14596ff66a7e
Jun 19 16:41:36 dist-test-slave-03cd krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887296, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:41:36.881140 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:36.865797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42299 (local address 127.0.98.62:43729)
0619 16:41:36.865978 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:36.865982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:36.867008 (+ 1026us) server_negotiation.cc:407] Connection header received
0619 16:41:36.868766 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:36.868770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:36.868841 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:36.868937 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:36.870750 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:36.871448 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:36.872914 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:36.873151 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:36.873276 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:36.876541 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:36.876545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:36.876547 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:36.876579 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:36.878295 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:36.878994 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:36.878998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:36.879000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:36.879101 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:36.879628 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:36.879632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:36.879634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:36.879962 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:36.880088 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:36.880762 (+ 674us) server_negotiation.cc:299] Negotiation successful
0619 16:41:36.880938 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:41:36.882534 6962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:36.866189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.98.3:42299)
0619 16:41:36.866797 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:36.866855 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:36.868484 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:36.869178 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:36.869188 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:36.869845 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:36.870580 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:36.870601 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:36.871597 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:36.871600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:36.872772 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:36.872784 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:36.873333 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:36.873336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:36.873543 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:36.874306 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:36.874326 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:36.876297 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:36.878507 (+ 2210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:36.878520 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:36.878541 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:36.878864 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:36.879208 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:36.879211 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:36.879213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:36.879481 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:36.880087 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:36.880097 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:36.880458 (+ 361us) client_negotiation.cc:769] Sending connection context
0619 16:41:36.880697 (+ 239us) client_negotiation.cc:240] Negotiation successful
0619 16:41:36.881160 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":134,"threads_started":1}
I20260619 16:41:36.884205 6960 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43729
I20260619 16:41:36.884590 6960 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:36.885383 6960 heartbeater.cc:511] Master 127.0.98.62:43729 requested a full tablet report, sending...
I20260619 16:41:36.886848 6501 ts_manager.cc:194] Registered new tserver with Master: a22a1178390044a9a7cc957008be869f (127.0.98.3:34813)
I20260619 16:41:36.887539 6501 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42299
I20260619 16:41:36.898205 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260619 16:41:36.932469 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46879
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43729,127.0.98.61:46879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:37.054834 6965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:37.055241 6965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:37.055330 6965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:37.059943 6965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:41:37.060034 6965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:37.060070 6965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:37.060133 6965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:41:37.060171 6965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:41:37.066205 6965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43729,127.0.98.61:46879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
W20260619 16:41:37.066531 6965 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:43729,127.0.98.61:46879'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 16:41:37.067521 6965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:37.068651 6965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:37.078898 6971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:37.078904 6970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:37.078961 6965 server_base.cc:1034] running on GCE node
W20260619 16:41:37.078895 6973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:37.079872 6965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:37.081113 6965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:37.082345 6965 hybrid_clock.cc:648] HybridClock initialized: now 1781887297082310 us; error 50 us; skew 500 ppm
Jun 19 16:41:37 dist-test-slave-03cd krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887297, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:37.085763 6965 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260619 16:41:37.086917 6965 webserver.cc:460] Webserver started at http://127.0.98.61:33345/ using document root <none> and password file <none>
I20260619 16:41:37.087563 6965 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:37.087695 6965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:37.087954 6965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:37.090322 6965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-1/data/instance:
uuid: "fb0fc19974204bc5aeb5f9e3f065d834"
format_stamp: "Formatted at 2026-06-19 16:41:37 on dist-test-slave-03cd"
server_key: "3b0d4236530c90fdd535adf713c59b71"
server_key_iv: "b2bf75f944294953e365968d5dd3071f"
server_key_version: "encryptionkey@0"
I20260619 16:41:37.090968 6965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-1/wal/instance:
uuid: "fb0fc19974204bc5aeb5f9e3f065d834"
format_stamp: "Formatted at 2026-06-19 16:41:37 on dist-test-slave-03cd"
server_key: "3b0d4236530c90fdd535adf713c59b71"
server_key_iv: "b2bf75f944294953e365968d5dd3071f"
server_key_version: "encryptionkey@0"
I20260619 16:41:37.095319 6965 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:41:37.098323 6980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:37.099968 6965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 16:41:37.100142 6965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-1/wal
uuid: "fb0fc19974204bc5aeb5f9e3f065d834"
format_stamp: "Formatted at 2026-06-19 16:41:37 on dist-test-slave-03cd"
server_key: "3b0d4236530c90fdd535adf713c59b71"
server_key_iv: "b2bf75f944294953e365968d5dd3071f"
server_key_version: "encryptionkey@0"
I20260619 16:41:37.100283 6965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:37.124558 6965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:37.128276 6965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:37.128487 6965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:37.135815 6965 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46879
I20260619 16:41:37.136111 7032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46879 every 8 connection(s)
I20260619 16:41:37.136952 6965 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-1/data/info.pb
I20260619 16:41:37.139955 7033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:37.140823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 6965
I20260619 16:41:37.144189 7033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:41:37 dist-test-slave-03cd krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 19 16:41:37 dist-test-slave-03cd krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887297, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:41:37.156342 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.146334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40276 (local address 127.0.98.62:43729)
0619 16:41:37.146494 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:37.146499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:37.146841 (+ 342us) server_negotiation.cc:407] Connection header received
0619 16:41:37.147014 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:37.147017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:37.147095 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:37.147182 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:37.148014 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.148512 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.149631 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.150031 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.150153 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.152544 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:37.152548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:37.152553 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:37.152581 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:37.154466 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.154942 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.154945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.154947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.155028 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.155301 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.155306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.155308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.155520 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:37.155626 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:37.156025 (+ 399us) server_negotiation.cc:299] Negotiation successful
0619 16:41:37.156147 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:41:37.156834 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.146253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.0.1:40276)
0619 16:41:37.146677 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:37.146760 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:37.146906 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:37.147284 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:37.147289 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:37.147628 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:37.147865 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.147871 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.148672 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.148675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.149516 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.149525 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.150196 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.150199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.150277 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.150807 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:37.150823 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:37.152360 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:37.154596 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.154604 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.154611 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.154839 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.155120 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.155125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.155127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.155210 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.155625 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:37.155634 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:37.155813 (+ 179us) client_negotiation.cc:769] Sending connection context
0619 16:41:37.155973 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 16:41:37.156157 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":157,"threads_started":1}
I20260619 16:41:37.156840 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.142615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40510 (local address 127.0.98.61:46879)
0619 16:41:37.143052 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:37.143069 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:37.143108 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:41:37.143842 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:37.143862 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:37.144233 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:37.144520 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:37.145498 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.147009 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.147959 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.148481 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.148639 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.151443 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:37.151450 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:37.151463 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:37.151492 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:37.153509 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.154006 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.154010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.154016 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.154122 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.154563 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.154570 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.154572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.155014 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:37.155241 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:37.155556 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 16:41:37.155880 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":153,"threads_started":1}
I20260619 16:41:37.158190 7033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:41:37 dist-test-slave-03cd krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887297, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 16:41:37.168241 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.159019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40516 (local address 127.0.98.61:46879)
0619 16:41:37.159195 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:37.159200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:37.159233 (+ 33us) server_negotiation.cc:407] Connection header received
0619 16:41:37.159269 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:37.159275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:37.159418 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:37.159548 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:37.160163 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.161242 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.162096 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.162362 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.162504 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.165037 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:37.165041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:37.165044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:37.165083 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:37.166586 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.167031 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.167037 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.167039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.167112 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.167378 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.167383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.167385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.167566 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:37.167710 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:37.167932 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 16:41:37.168100 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":102}
I20260619 16:41:37.168241 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.158774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46879 (local address 127.0.0.1:40516)
0619 16:41:37.158972 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:37.158993 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:37.159195 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:37.159603 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:37.159606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:37.159881 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:37.159991 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.159998 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.161372 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.161378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.161951 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.161959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.162504 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.162508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.162630 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.163192 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:37.163206 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:37.164873 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:37.166740 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.166744 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.166747 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.166936 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.167207 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.167211 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.167213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.167292 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.167682 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:37.167685 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:37.167768 (+ 83us) client_negotiation.cc:769] Sending connection context
0619 16:41:37.167891 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 16:41:37.168043 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260619 16:41:37.171577 7033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834: Bootstrap starting.
I20260619 16:41:37.174062 7033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:37.174899 7033 log.cc:824] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:37.177237 7033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834: No bootstrap required, opened a new log
I20260619 16:41:37.180603 7033 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } }
I20260619 16:41:37.180856 7033 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:37.180891 7033 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0fc19974204bc5aeb5f9e3f065d834, State: Initialized, Role: FOLLOWER
I20260619 16:41:37.181372 7033 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } }
I20260619 16:41:37.181875 7039 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } } }
I20260619 16:41:37.182026 7039 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:41:37.182787 7033 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 16:41:37.188560 7050 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:41:37.188681 7050 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:41:37.191078 6965 master_runner.cc:428] Detected that this master fb0fc19974204bc5aeb5f9e3f065d834 is joining an existing cluster
I20260619 16:41:37.191179 6965 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:46879
I20260619 16:41:37.202312 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.193791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40524 (local address 127.0.98.61:46879)
0619 16:41:37.193898 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:37.193903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:37.193918 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:41:37.193964 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:37.193966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:37.194015 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:37.194111 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:37.194788 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.195634 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.196750 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.197127 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.197260 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.198790 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:37.198794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:37.198796 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:37.198816 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:37.200620 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.201114 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.201117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.201119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.201211 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.201468 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.201472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.201474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.201659 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:37.201764 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:37.202008 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 16:41:37.202169 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":43}
I20260619 16:41:37.202610 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.193399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40288 (local address 127.0.98.62:43729)
0619 16:41:37.193532 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:37.193537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:37.193774 (+ 237us) server_negotiation.cc:407] Connection header received
0619 16:41:37.193974 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:37.193977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:37.194016 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:37.194059 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:37.194792 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.195336 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.196448 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.196799 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.196909 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.198797 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:37.198801 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:37.198804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:37.198834 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:37.200947 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.201379 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.201383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.201385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.201460 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.201745 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.201748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.201750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.201951 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:37.202016 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:37.202340 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 16:41:37.202455 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:41:37.202320 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.193456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46879 (local address 127.0.0.1:40524)
0619 16:41:37.193780 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:37.193794 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:37.193856 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:37.194143 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:37.194146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:37.194426 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:37.194648 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.194656 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.195916 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.195919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.196644 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.196651 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.197376 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.197381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.197471 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.197944 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:37.197964 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:37.198642 (+ 678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:37.200761 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.200764 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.200766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.201001 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.201303 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.201307 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.201309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.201396 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.201753 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:37.201756 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:37.201846 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 16:41:37.201957 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 16:41:37.202113 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":23,"thread_start_us":96,"threads_started":1}
I20260619 16:41:37.203125 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.193346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.0.1:40288)
0619 16:41:37.193689 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:37.193705 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:37.193855 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:37.194243 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:37.194246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:37.194458 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:37.194647 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.194656 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.195458 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.195464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.196326 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.196335 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.196937 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.196940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.197135 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.198267 (+ 1132us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:37.198281 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:37.198671 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:37.201063 (+ 2392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.201067 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.201071 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.201290 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.201556 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.201559 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.201561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.201638 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.202085 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:37.202087 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:37.202170 (+ 83us) client_negotiation.cc:769] Sending connection context
0619 16:41:37.202807 (+ 637us) client_negotiation.cc:240] Negotiation successful
0619 16:41:37.202952 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":41,"thread_start_us":133,"threads_started":1}
Jun 19 16:41:37 dist-test-slave-03cd krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887295, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260619 16:41:37.220216 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.208644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46879 (local address 127.0.0.1:40532)
0619 16:41:37.209003 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:37.209050 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:37.209184 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:37.209547 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:37.209552 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:37.209947 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:37.210105 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.210112 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.211137 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.211140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.212180 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.212191 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.212845 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.212849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.212963 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.213639 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:37.213655 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:37.216027 (+ 2372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:37.218237 (+ 2210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.218253 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.218264 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.218626 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.218967 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.218971 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.218973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.219069 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.219534 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:37.219545 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:37.219768 (+ 223us) client_negotiation.cc:769] Sending connection context
0619 16:41:37.219903 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 16:41:37.220073 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":101,"threads_started":1}
I20260619 16:41:37.220263 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.208694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40532 (local address 127.0.98.61:46879)
0619 16:41:37.208831 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:37.208836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:37.209127 (+ 291us) server_negotiation.cc:407] Connection header received
0619 16:41:37.209321 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:37.209325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:37.209379 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:37.209468 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:37.210224 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.211004 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.212323 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.212714 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.212841 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.216224 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:37.216228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:37.216231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:37.216264 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:37.218083 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.218765 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.218768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.218770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.218859 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.219174 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.219177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.219179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.219412 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:37.219511 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:37.219979 (+ 468us) server_negotiation.cc:299] Negotiation successful
0619 16:41:37.220104 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:41:37.221966 6501 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:46879: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 46879
}
attrs {
promote: true
}
}
dest_uuid: "d328d7aca0674111beef001a9009b0f8"
cas_config_opid_index: -1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260619 16:41:37.222687 6501 consensus_queue.cc:237] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: true } }
I20260619 16:41:37.224575 7062 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER fb0fc19974204bc5aeb5f9e3f065d834 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: true } } }
I20260619 16:41:37.225303 7007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:37.226361 7062 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:46879
I20260619 16:41:37.226615 7062 sys_catalog.cc:455] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d328d7aca0674111beef001a9009b0f8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: true } } }
I20260619 16:41:37.226722 7062 sys_catalog.cc:458] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:37.226807 7007 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [term 1 FOLLOWER]: Refusing update from remote peer d328d7aca0674111beef001a9009b0f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 16:41:37.226878 6965 master.cc:537] Master@127.0.98.61:46879 shutting down...
I20260619 16:41:37.227604 7062 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:41:37.230162 6487 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260619 16:41:37.231060 6487 consensus_peers.cc:487] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 -> Peer fb0fc19974204bc5aeb5f9e3f065d834 (127.0.98.61:46879): Couldn't send request to peer fb0fc19974204bc5aeb5f9e3f065d834. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 16:41:37.231680 6965 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:41:37.231966 6965 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:37.232090 6965 tablet_replica.cc:330] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834: stopping tablet replica
I20260619 16:41:37.237071 6965 master.cc:552] Master@127.0.98.61:46879 shutdown complete.
I20260619 16:41:37.237125 6965 master_runner.cc:305] Clearing existing system tablet
I20260619 16:41:37.237402 6965 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:37.237452 6965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:37.239455 6965 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:41:37.240558 7067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:37.240813 6965 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:37.240934 6965 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:41:37.241043 6965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-1/wal
uuid: "fb0fc19974204bc5aeb5f9e3f065d834"
format_stamp: "Formatted at 2026-06-19 16:41:37 on dist-test-slave-03cd"
server_key: "3b0d4236530c90fdd535adf713c59b71"
server_key_iv: "b2bf75f944294953e365968d5dd3071f"
server_key_version: "encryptionkey@0"
I20260619 16:41:37.243558 6965 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:37.247095 6965 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 16:41:37.247216 6965 log.cc:1196] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 16:41:37.247500 6965 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834: Deleting consensus metadata
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260619 16:41:37.247716 6965 master_runner.cc:315] Copying system tablet from 127.0.98.62:43729
I20260619 16:41:37.248621 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43141
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43729,127.0.98.61:46879,127.0.98.60:43141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:41:37.248982 6965 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:43729
I20260619 16:41:37.258073 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.249731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40302 (local address 127.0.98.62:43729)
0619 16:41:37.249861 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:37.249866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:37.250141 (+ 275us) server_negotiation.cc:407] Connection header received
0619 16:41:37.250378 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:37.250382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:37.250439 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:37.250542 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:37.251040 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.251627 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.252587 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.252831 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.252953 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.254386 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:37.254390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:37.254393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:37.254427 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:37.256364 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.256831 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.256837 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.256838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.256919 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.257193 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.257198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.257200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.257406 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:37.257511 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:37.257776 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 16:41:37.257886 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:41:37.258105 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.249629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.0.1:40302)
0619 16:41:37.250022 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:37.250044 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:37.250251 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:37.250562 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:37.250566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:37.250799 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:37.250904 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.250911 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.251781 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.251786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.252438 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.252448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.252961 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.252965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.253068 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.253828 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:37.253841 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:37.254201 (+ 360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:37.256491 (+ 2290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.256496 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.256499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.256718 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.257011 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.257015 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.257017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.257102 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.257500 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:37.257503 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:37.257608 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 16:41:37.257716 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 16:41:37.257889 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":103,"threads_started":1}
I20260619 16:41:37.258697 6521 tablet_copy_service.cc:140] P d328d7aca0674111beef001a9009b0f8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fb0fc19974204bc5aeb5f9e3f065d834 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:40302)
I20260619 16:41:37.258903 6521 tablet_copy_service.cc:161] P d328d7aca0674111beef001a9009b0f8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fb0fc19974204bc5aeb5f9e3f065d834 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:40302: session id = fb0fc19974204bc5aeb5f9e3f065d834-00000000000000000000000000000000
I20260619 16:41:37.263181 6521 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 16:41:37.266147 6965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 16:41:37.269479 6965 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834: tablet copy: Starting download of 0 data blocks...
I20260619 16:41:37.269714 6965 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834: tablet copy: Starting download of 1 WAL segments...
I20260619 16:41:37.271618 6965 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 16:41:37.273555 6521 tablet_copy_service.cc:342] P d328d7aca0674111beef001a9009b0f8: Request end of tablet copy session fb0fc19974204bc5aeb5f9e3f065d834-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:40302
I20260619 16:41:37.273699 6521 tablet_copy_service.cc:433] P d328d7aca0674111beef001a9009b0f8: ending tablet copy session fb0fc19974204bc5aeb5f9e3f065d834-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fb0fc19974204bc5aeb5f9e3f065d834
I20260619 16:41:37.281584 6965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:37.283610 7076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:37.283610 7075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:37.284078 7078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:37.285777 6965 server_base.cc:1034] running on GCE node
I20260619 16:41:37.286007 6965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:37.286377 6965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:37.287555 6965 hybrid_clock.cc:648] HybridClock initialized: now 1781887297287586 us; error 69 us; skew 500 ppm
Jun 19 16:41:37 dist-test-slave-03cd krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887297, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:37.290249 6965 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260619 16:41:37.290808 6965 webserver.cc:460] Webserver started at http://127.0.98.61:35969/ using document root <none> and password file <none>
I20260619 16:41:37.291015 6965 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:37.291055 6965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:37.296150 6965 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.000s
I20260619 16:41:37.297238 7085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:37.297502 6965 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:41:37.297605 6965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-1/wal
uuid: "fb0fc19974204bc5aeb5f9e3f065d834"
format_stamp: "Formatted at 2026-06-19 16:41:37 on dist-test-slave-03cd"
server_key: "3b0d4236530c90fdd535adf713c59b71"
server_key_iv: "b2bf75f944294953e365968d5dd3071f"
server_key_version: "encryptionkey@0"
I20260619 16:41:37.297720 6965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:37.312937 6965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:37.315366 6965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:37.320818 6965 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46879
I20260619 16:41:37.321527 6965 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-1/data/info.pb
I20260619 16:41:37.321978 7137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46879 every 8 connection(s)
I20260619 16:41:37.323140 7138 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:41:37.324085 7138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834: Bootstrap starting.
I20260619 16:41:37.339411 7138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:41:37.339869 7138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834: Bootstrap complete.
I20260619 16:41:37.340538 7138 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: true } }
I20260619 16:41:37.340718 7138 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fb0fc19974204bc5aeb5f9e3f065d834, State: Initialized, Role: LEARNER
I20260619 16:41:37.340868 7138 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: true } }
I20260619 16:41:37.341291 7141 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: true } } }
I20260619 16:41:37.341470 7141 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [sys.catalog]: This master's current role is: LEARNER
I20260619 16:41:37.341513 7138 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:41:37.345018 7152 catalog_manager.cc:1259] Loaded cluster ID: 02c7ab1904fc44fdb1154a0c0059f3af
I20260619 16:41:37.345140 7152 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834: loading cluster ID for follower catalog manager: success
I20260619 16:41:37.346769 7152 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834: acquiring CA information for follower catalog manager: success
I20260619 16:41:37.348091 7152 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:41:37.359995 6692 heartbeater.cc:503] Master 127.0.98.62:43729 was elected leader, sending a full tablet report...
W20260619 16:41:37.424685 7074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:37.425168 7074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:37.425279 7074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:37.429731 7074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:41:37.429836 7074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:37.429874 7074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:37.429934 7074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:41:37.429973 7074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:41:37.435815 7074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43729,127.0.98.61:46879,127.0.98.60:43141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:37.437052 7074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:37.438133 7074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:37.448663 7157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:37.448706 7074 server_base.cc:1034] running on GCE node
W20260619 16:41:37.448663 7158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:37.448668 7160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:37.449597 7074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:37.450809 7074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:37.452021 7074 hybrid_clock.cc:648] HybridClock initialized: now 1781887297452008 us; error 30 us; skew 500 ppm
Jun 19 16:41:37 dist-test-slave-03cd krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887297, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:37.455061 7074 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260619 16:41:37.456278 7074 webserver.cc:460] Webserver started at http://127.0.98.60:33117/ using document root <none> and password file <none>
I20260619 16:41:37.456960 7074 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:37.457032 7074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:37.457374 7074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:37.459849 7074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-2/data/instance:
uuid: "3b17354865b2491fb270179636349ab4"
format_stamp: "Formatted at 2026-06-19 16:41:37 on dist-test-slave-03cd"
server_key: "82f4c752c621c1570f01411a2b98177d"
server_key_iv: "50db41f696903fb97f982ff9637eca61"
server_key_version: "encryptionkey@0"
I20260619 16:41:37.460458 7074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-2/wal/instance:
uuid: "3b17354865b2491fb270179636349ab4"
format_stamp: "Formatted at 2026-06-19 16:41:37 on dist-test-slave-03cd"
server_key: "82f4c752c621c1570f01411a2b98177d"
server_key_iv: "50db41f696903fb97f982ff9637eca61"
server_key_version: "encryptionkey@0"
I20260619 16:41:37.465270 7074 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:41:37.468003 7167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:37.469491 7074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:41:37.469678 7074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-2/wal
uuid: "3b17354865b2491fb270179636349ab4"
format_stamp: "Formatted at 2026-06-19 16:41:37 on dist-test-slave-03cd"
server_key: "82f4c752c621c1570f01411a2b98177d"
server_key_iv: "50db41f696903fb97f982ff9637eca61"
server_key_version: "encryptionkey@0"
I20260619 16:41:37.469846 7074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:37.486493 7074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:37.490337 7074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:37.490577 7074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:37.497857 7074 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43141
I20260619 16:41:37.498253 7219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43141 every 8 connection(s)
I20260619 16:41:37.498967 7074 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-2/data/info.pb
I20260619 16:41:37.501926 7220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:37.506342 7220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:41:37.506734 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 7074
Jun 19 16:41:37 dist-test-slave-03cd krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 16:41:37 dist-test-slave-03cd krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887297, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:41:37.520356 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.508635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40312 (local address 127.0.98.62:43729)
0619 16:41:37.508789 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:37.508793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:37.509179 (+ 386us) server_negotiation.cc:407] Connection header received
0619 16:41:37.509918 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:37.509921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:37.509972 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:37.510051 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:37.511111 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.511805 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.512879 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.513119 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.513215 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.516790 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:37.516793 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:37.516795 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:37.516821 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:37.518277 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.518822 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.518826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.518834 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.518930 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.519260 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.519265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.519267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.519470 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:37.519579 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:37.520047 (+ 468us) server_negotiation.cc:299] Negotiation successful
0619 16:41:37.520174 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 16:41:37.521245 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.508360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36472 (local address 127.0.98.60:43141)
0619 16:41:37.508845 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:37.508862 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:37.508902 (+ 40us) server_negotiation.cc:407] Connection header received
0619 16:41:37.509660 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:37.509678 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:37.509908 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:37.510089 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:37.510847 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.511838 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.512822 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.513214 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.513371 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.516286 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:37.516293 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:37.516309 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:37.516339 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:37.518358 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.518895 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.518901 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.518902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.519006 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.519284 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.519289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.519291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.519741 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:37.519947 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:37.520189 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 16:41:37.520402 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":120,"server-negotiator.queue_time_us":363,"thread_start_us":182,"threads_started":1}
I20260619 16:41:37.521242 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.508413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.0.1:40312)
0619 16:41:37.509049 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:37.509096 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:37.509686 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:37.510206 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:37.510211 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:37.510614 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:37.510930 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.510942 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.511947 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.511952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.512713 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.512721 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.513273 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.513279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.513491 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.514320 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:37.514336 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:37.516622 (+ 2286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:37.518447 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.518456 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.518467 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.518709 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.519035 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.519039 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.519041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.519164 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.519576 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:37.519584 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:37.519807 (+ 223us) client_negotiation.cc:769] Sending connection context
0619 16:41:37.519992 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 16:41:37.520336 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":91,"spinlock_wait_cycles":116992,"thread_start_us":209,"threads_started":1}
I20260619 16:41:37.522614 7220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:41:37 dist-test-slave-03cd krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887297, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 16:41:37.531446 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.523169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46879 (local address 127.0.0.1:40536)
0619 16:41:37.523288 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:37.523302 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:37.523405 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:37.523880 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:37.523882 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:37.524036 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:37.524134 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.524140 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.525045 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.525048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.525686 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.525694 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.526143 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.526146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.526225 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.526685 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:37.526696 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:37.528034 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:37.529879 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.529882 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.529883 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.530099 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.530445 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.530447 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.530449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.530533 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.530952 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:37.530955 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:37.531051 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 16:41:37.531170 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 16:41:37.531302 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 16:41:37.531479 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.523275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40536 (local address 127.0.98.61:46879)
0619 16:41:37.523607 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:37.523612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:37.523625 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:41:37.523719 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:37.523722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:37.523773 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:37.523850 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:37.524238 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.524939 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.525790 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.526018 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.526102 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.528192 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:37.528195 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:37.528197 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:37.528220 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:37.529744 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.530224 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.530227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.530229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.530310 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.530625 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.530628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.530629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.530819 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:37.530898 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:37.531234 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 16:41:37.531343 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":130,"threads_started":1}
I20260619 16:41:37.532536 7220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43141 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:41:37 dist-test-slave-03cd krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887297, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 16:41:37.541453 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.533152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43141 (local address 127.0.0.1:36484)
0619 16:41:37.533326 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:37.533344 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:37.533445 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:37.533714 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:37.533718 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:37.533949 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:37.534023 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.534028 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.534858 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.534865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.535425 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.535433 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.536002 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.536005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.536076 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.536609 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:37.536622 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:37.537987 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:37.539629 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.539632 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.539635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.539887 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.540235 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.540239 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.540241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.540336 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.540758 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:37.540761 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:37.540863 (+ 102us) client_negotiation.cc:769] Sending connection context
0619 16:41:37.540983 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 16:41:37.541152 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 16:41:37.541456 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.533193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36484 (local address 127.0.98.60:43141)
0619 16:41:37.533326 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:37.533333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:37.533399 (+ 66us) server_negotiation.cc:407] Connection header received
0619 16:41:37.533566 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:37.533574 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:37.533619 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:37.533707 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:37.534150 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.534739 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.535552 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.535881 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.536002 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.538129 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:37.538133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:37.538138 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:37.538162 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:37.539482 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.539993 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.540000 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.540002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.540130 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.540445 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.540448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.540450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.540647 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:37.540759 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:37.541040 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 16:41:37.541215 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":48}
I20260619 16:41:37.545553 7220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4: Bootstrap starting.
I20260619 16:41:37.548056 7220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:37.548831 7220 log.cc:824] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:37.550879 7220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4: No bootstrap required, opened a new log
I20260619 16:41:37.554076 7220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43141 } }
I20260619 16:41:37.554356 7220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:37.554468 7220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b17354865b2491fb270179636349ab4, State: Initialized, Role: FOLLOWER
I20260619 16:41:37.554946 7220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43141 } }
I20260619 16:41:37.555497 7226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43141 } } }
I20260619 16:41:37.555617 7226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:41:37.556265 7220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 16:41:37.562124 7237 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:41:37.562247 7237 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:41:37.566031 7074 master_runner.cc:428] Detected that this master 3b17354865b2491fb270179636349ab4 is joining an existing cluster
I20260619 16:41:37.566141 7074 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:43141
I20260619 16:41:37.575152 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 6563
I20260619 16:41:37.581200 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41841
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43729
--webserver_port=42149
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43729,127.0.98.61:46879
--tserver_master_addrs=127.0.98.62:43729,127.0.98.61:46879,127.0.98.60:43141 with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:41:37.585664 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.569188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40320 (local address 127.0.98.62:43729)
0619 16:41:37.569320 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:37.569325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:37.569880 (+ 555us) server_negotiation.cc:407] Connection header received
0619 16:41:37.570281 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:37.570287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:37.570352 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:37.570456 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:37.571153 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.572117 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.573530 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.573893 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.573971 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.580876 (+ 6905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:37.580881 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:37.580884 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:37.580924 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:37.583042 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.583775 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.583781 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.583785 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.583888 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.584293 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.584299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.584305 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.584592 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:37.584680 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:37.585249 (+ 569us) server_negotiation.cc:299] Negotiation successful
0619 16:41:37.585406 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:41:37.586273 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.568926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.0.1:40320)
0619 16:41:37.569792 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:37.569802 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:37.569992 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:37.570625 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:37.570629 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:37.570870 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:37.571008 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.571019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.572260 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.572263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.573356 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.573367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.574147 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.574151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.574286 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.574791 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:37.574806 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:37.580649 (+ 5843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:37.583239 (+ 2590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.583245 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.583249 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.583580 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.584048 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.584053 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.584055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.584171 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.584763 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:37.584769 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:37.584914 (+ 145us) client_negotiation.cc:769] Sending connection context
0619 16:41:37.585907 (+ 993us) client_negotiation.cc:240] Negotiation successful
0619 16:41:37.586086 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":84,"thread_start_us":91,"threads_started":1}
I20260619 16:41:37.589322 7226 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:41:37.589696 7226 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43141 } }
I20260619 16:41:37.590922 7226 leader_election.cc:288] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d328d7aca0674111beef001a9009b0f8 (127.0.98.62:43729), fb0fc19974204bc5aeb5f9e3f065d834 (127.0.98.61:46879)
I20260619 16:41:37.591300 6511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b17354865b2491fb270179636349ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d328d7aca0674111beef001a9009b0f8" is_pre_election: true
I20260619 16:41:37.591850 7112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b17354865b2491fb270179636349ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" is_pre_election: true
I20260619 16:41:37.592018 7112 raft_consensus.cc:1799] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [term 1 LEARNER]: Handling vote request from an unknown peer 3b17354865b2491fb270179636349ab4
I20260619 16:41:37.592144 7112 raft_consensus.cc:2396] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 3b17354865b2491fb270179636349ab4 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260619 16:41:37.592890 7169 leader_election.cc:302] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b17354865b2491fb270179636349ab4; no voters: d328d7aca0674111beef001a9009b0f8, fb0fc19974204bc5aeb5f9e3f065d834
I20260619 16:41:37.593477 7226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:37.596097 7226 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 16:41:37.598030 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.569054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36488 (local address 127.0.98.60:43141)
0619 16:41:37.569158 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:37.569163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:37.569492 (+ 329us) server_negotiation.cc:407] Connection header received
0619 16:41:37.569707 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:37.569710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:37.569762 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:37.569963 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:37.570659 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.571425 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.572743 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.573062 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.573167 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.594222 (+ 21055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:37.594226 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:37.594229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:37.594254 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:37.596122 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.596698 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.596701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.596703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.596772 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.597024 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.597027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.597028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.597239 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:37.597304 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:37.597720 (+ 416us) server_negotiation.cc:299] Negotiation successful
0619 16:41:37.597871 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 16:41:37.598896 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.568919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43141 (local address 127.0.0.1:36488)
0619 16:41:37.569417 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:37.569432 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:37.569578 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:37.570119 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:37.570122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:37.570387 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:37.570535 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.570543 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.571547 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.571551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.572601 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.572611 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.573340 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.573343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.573516 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.574295 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:37.574309 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:37.594066 (+ 19757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:37.596326 (+ 2260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.596329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.596331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.596594 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.596860 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.596862 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.596864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.596933 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.597413 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:37.597416 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:37.597515 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 16:41:37.598517 (+ 1002us) client_negotiation.cc:240] Negotiation successful
0619 16:41:37.598666 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":52,"spinlock_wait_cycles":8448,"thread_start_us":238,"threads_started":1}
I20260619 16:41:37.601609 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.569518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40540 (local address 127.0.98.61:46879)
0619 16:41:37.569631 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:37.569635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:37.570559 (+ 924us) server_negotiation.cc:407] Connection header received
0619 16:41:37.570750 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:37.570757 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:37.570816 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:37.570873 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:37.571545 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.572394 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.573673 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.574069 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.574150 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.597407 (+ 23257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:37.597412 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:37.597415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:37.597448 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:37.599288 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.600004 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.600008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.600010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.600101 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.600388 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.600391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.600393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.600602 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:37.600670 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:37.601283 (+ 613us) server_negotiation.cc:299] Negotiation successful
0619 16:41:37.601438 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 16:41:37.602205 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.569485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46879 (local address 127.0.0.1:40540)
0619 16:41:37.570470 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:37.570485 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:37.570625 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:37.571064 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:37.571070 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:37.571274 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:37.571395 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.571403 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.572551 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.572554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.573551 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.573559 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.574302 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.574305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.574407 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.596737 (+ 22330us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:37.596754 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:37.597209 (+ 455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:37.599438 (+ 2229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.599441 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.599444 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.599866 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.600212 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.600215 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.600217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.600298 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.600741 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:37.600744 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:37.600845 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 16:41:37.601809 (+ 964us) client_negotiation.cc:240] Negotiation successful
0619 16:41:37.602019 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"mutex_wait_us":37,"thread_start_us":104,"threads_started":1}
Jun 19 16:41:37 dist-test-slave-03cd krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887295, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 16:41:37.610462 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.592448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36494 (local address 127.0.98.60:43141)
0619 16:41:37.597564 (+ 5116us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:37.597575 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:37.597589 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:41:37.597669 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:37.597674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:37.597732 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:37.599063 (+ 1331us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:37.599174 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.600789 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.602116 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.602511 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.603169 (+ 658us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.606217 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:37.606222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:37.606225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:37.606253 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:37.608470 (+ 2217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.609050 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.609053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.609054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.609138 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.609427 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.609429 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.609431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.609673 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:37.609775 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:37.610140 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 16:41:37.610274 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5033,"thread_start_us":119,"threads_started":1}
I20260619 16:41:37.610514 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.591789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43141 (local address 127.0.0.1:36494)
0619 16:41:37.592171 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:37.592190 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:37.592282 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:37.598577 (+ 6295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:37.598581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:37.598782 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:37.598915 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.598925 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.601078 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.601083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.601952 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.601965 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.602758 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.602763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.602881 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.604288 (+ 1407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:37.604305 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:37.605918 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:37.608631 (+ 2713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.608635 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.608637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.608900 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.609237 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.609240 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.609242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.609323 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.609815 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:37.609818 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:37.609948 (+ 130us) client_negotiation.cc:769] Sending connection context
0619 16:41:37.610062 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 16:41:37.610234 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272}
I20260619 16:41:37.611853 6501 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:43141: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3b17354865b2491fb270179636349ab4"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 43141
}
attrs {
promote: true
}
}
dest_uuid: "d328d7aca0674111beef001a9009b0f8"
cas_config_opid_index: 5
I20260619 16:41:37.612571 6501 consensus_queue.cc:237] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: true } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: true } }
I20260619 16:41:37.613721 7062 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3b17354865b2491fb270179636349ab4 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: true } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: true } } }
I20260619 16:41:37.615335 7062 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:43141
I20260619 16:41:37.615528 7062 sys_catalog.cc:455] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d328d7aca0674111beef001a9009b0f8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: true } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: true } } }
I20260619 16:41:37.615630 7062 sys_catalog.cc:458] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:37.616961 7194 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [term 1 FOLLOWER]: Refusing update from remote peer d328d7aca0674111beef001a9009b0f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 16:41:37.617554 7063 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:37.619305 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.613885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46879 (local address 127.0.0.1:40542)
0619 16:41:37.614358 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:37.614375 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:37.614460 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:37.615074 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:37.615078 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:37.615094 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:37.615184 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.615190 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.616360 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.616364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.617602 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.617613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.618858 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.618862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.618966 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.619017 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 16:41:37.619092 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 16:41:37.619172 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376}
I20260619 16:41:37.619985 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.614129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40542 (local address 127.0.98.61:46879)
0619 16:41:37.614721 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:37.614726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:37.614740 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:41:37.614790 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:37.614793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:37.614849 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:37.614948 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:37.615303 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.616121 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.617863 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.618546 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.618650 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.618797 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:37.619772 (+ 975us) server_negotiation.cc:299] Negotiation successful
0619 16:41:37.619863 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510}
I20260619 16:41:37.620632 7112 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [term 1 LEARNER]: Refusing update from remote peer d328d7aca0674111beef001a9009b0f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 16:41:37.622243 7063 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 16:41:37.622848 7074 master.cc:537] Master@127.0.98.60:43141 shutting down...
I20260619 16:41:37.623528 7194 raft_consensus.cc:858] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43141 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: true } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: true } }
W20260619 16:41:37.623642 7194 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43141 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: true } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: true } }
I20260619 16:41:37.624383 7112 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3b17354865b2491fb270179636349ab4 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: true } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: true } } }
I20260619 16:41:37.625118 7194 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER fb0fc19974204bc5aeb5f9e3f065d834 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: true } } }
I20260619 16:41:37.626533 7141 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d328d7aca0674111beef001a9009b0f8. Latest consensus state: current_term: 1 leader_uuid: "d328d7aca0674111beef001a9009b0f8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: true } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: true } } }
I20260619 16:41:37.626647 7141 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [sys.catalog]: This master's current role is: LEARNER
I20260619 16:41:37.626718 7141 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d328d7aca0674111beef001a9009b0f8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: true } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: true } } }
I20260619 16:41:37.626778 7141 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [sys.catalog]: This master's current role is: LEARNER
I20260619 16:41:37.627812 7226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d328d7aca0674111beef001a9009b0f8. Latest consensus state: current_term: 1 leader_uuid: "d328d7aca0674111beef001a9009b0f8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: true } } }
I20260619 16:41:37.628013 7226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 16:41:37.628283 7226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d328d7aca0674111beef001a9009b0f8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: true } } }
I20260619 16:41:37.628485 7226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 16:41:37.628582 7226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d328d7aca0674111beef001a9009b0f8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: true } } }
I20260619 16:41:37.628643 7226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 16:41:37.631233 7063 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8: attempting to promote NON_VOTER fb0fc19974204bc5aeb5f9e3f065d834 to VOTER
I20260619 16:41:37.631767 7063 consensus_queue.cc:237] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: true } }
I20260619 16:41:37.632792 6826 heartbeater.cc:503] Master 127.0.98.62:43729 was elected leader, sending a full tablet report...
I20260619 16:41:37.635373 7112 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [term 1 LEARNER]: Refusing update from remote peer d328d7aca0674111beef001a9009b0f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260619 16:41:37.635390 6485 consensus_peers.cc:487] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 -> Peer 3b17354865b2491fb270179636349ab4 (127.0.98.60:43141): Couldn't send request to peer 3b17354865b2491fb270179636349ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 16:41:37.636076 7062 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 16:41:37.636742 7248 mvcc.cc:205] Tried to move back new op lower bound from 7298610364479021056 to 7298610364263559168. Current Snapshot: MvccSnapshot[applied={T|T < 7298610364311711744}]
I20260619 16:41:37.636919 7074 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260619 16:41:37.637176 7074 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260619 16:41:37.637265 7074 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4: stopping tablet replica
I20260619 16:41:37.639422 7255 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, fb0fc19974204bc5aeb5f9e3f065d834 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: true } } }
I20260619 16:41:37.641559 7112 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, fb0fc19974204bc5aeb5f9e3f065d834 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: true } } }
I20260619 16:41:37.643698 7252 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d328d7aca0674111beef001a9009b0f8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: true } } }
I20260619 16:41:37.643867 7252 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:41:37.645372 7063 sys_catalog.cc:455] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d328d7aca0674111beef001a9009b0f8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: true } } }
I20260619 16:41:37.645545 7063 sys_catalog.cc:458] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:37.645732 7255 sys_catalog.cc:455] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d328d7aca0674111beef001a9009b0f8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: true } } }
I20260619 16:41:37.645890 7255 sys_catalog.cc:458] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:37.663141 7074 master.cc:552] Master@127.0.98.60:43141 shutdown complete.
I20260619 16:41:37.663218 7074 master_runner.cc:305] Clearing existing system tablet
I20260619 16:41:37.663475 7074 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:37.663534 7074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:37.665045 7074 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:41:37.666010 7259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:37.666230 7074 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:37.666337 7074 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:41:37.666366 7074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-2/wal
uuid: "3b17354865b2491fb270179636349ab4"
format_stamp: "Formatted at 2026-06-19 16:41:37 on dist-test-slave-03cd"
server_key: "82f4c752c621c1570f01411a2b98177d"
server_key_iv: "50db41f696903fb97f982ff9637eca61"
server_key_version: "encryptionkey@0"
I20260619 16:41:37.668716 7074 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:37.671591 7074 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 16:41:37.671722 7074 log.cc:1196] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 16:41:37.672050 7074 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4: Deleting consensus metadata
I20260619 16:41:37.672220 7074 master_runner.cc:315] Copying system tablet from 127.0.98.62:43729
I20260619 16:41:37.673436 7074 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:43729
I20260619 16:41:37.683640 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.674077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.0.1:40322)
0619 16:41:37.674393 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:37.674408 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:37.674529 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:37.675060 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:37.675063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:37.675294 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:37.675435 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.675443 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.676632 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.676635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.677550 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.677561 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.678170 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.678173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.678274 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.678906 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:37.678922 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:37.679376 (+ 454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:37.681816 (+ 2440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.681820 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.681823 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.682123 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.682466 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.682470 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.682472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.682570 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.683050 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:37.683053 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:37.683167 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 16:41:37.683276 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 16:41:37.683444 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":95,"threads_started":1}
I20260619 16:41:37.684046 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.674197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40322 (local address 127.0.98.62:43729)
0619 16:41:37.674338 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:37.674343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:37.674499 (+ 156us) server_negotiation.cc:407] Connection header received
0619 16:41:37.674624 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:37.674627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:37.674684 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:37.674796 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:37.675581 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.676501 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.677684 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.678037 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.678156 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.679545 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:37.679549 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:37.679552 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:37.679582 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:37.681674 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.682244 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.682248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.682251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.682351 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.682681 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.682685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.682687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.682940 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:37.683053 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:37.683360 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 16:41:37.683505 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:41:37.684428 6521 tablet_copy_service.cc:140] P d328d7aca0674111beef001a9009b0f8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3b17354865b2491fb270179636349ab4 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:40322)
I20260619 16:41:37.684540 6521 tablet_copy_service.cc:161] P d328d7aca0674111beef001a9009b0f8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3b17354865b2491fb270179636349ab4 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:40322: session id = 3b17354865b2491fb270179636349ab4-00000000000000000000000000000000
I20260619 16:41:37.685762 6521 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 16:41:37.687283 7074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 16:41:37.690008 7074 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4: tablet copy: Starting download of 0 data blocks...
I20260619 16:41:37.690229 7074 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4: tablet copy: Starting download of 1 WAL segments...
I20260619 16:41:37.692211 7074 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 16:41:37.694438 6521 tablet_copy_service.cc:342] P d328d7aca0674111beef001a9009b0f8: Request end of tablet copy session 3b17354865b2491fb270179636349ab4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:40322
I20260619 16:41:37.694554 6521 tablet_copy_service.cc:433] P d328d7aca0674111beef001a9009b0f8: ending tablet copy session 3b17354865b2491fb270179636349ab4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3b17354865b2491fb270179636349ab4
I20260619 16:41:37.702682 7074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:37.704815 7266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:37.704999 7269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:37.705274 7267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:37.707865 7074 server_base.cc:1034] running on GCE node
I20260619 16:41:37.708110 7074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:37.708489 7074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:37.709645 7074 hybrid_clock.cc:648] HybridClock initialized: now 1781887297709624 us; error 48 us; skew 500 ppm
Jun 19 16:41:37 dist-test-slave-03cd krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887297, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:37.711444 7074 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260619 16:41:37.712040 7074 webserver.cc:460] Webserver started at http://127.0.98.60:34143/ using document root <none> and password file <none>
I20260619 16:41:37.712242 7074 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:37.712287 7074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:37.713897 7074 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:41:37.715741 7276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:37.716118 7074 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:41:37.716210 7074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-2/wal
uuid: "3b17354865b2491fb270179636349ab4"
format_stamp: "Formatted at 2026-06-19 16:41:37 on dist-test-slave-03cd"
server_key: "82f4c752c621c1570f01411a2b98177d"
server_key_iv: "50db41f696903fb97f982ff9637eca61"
server_key_version: "encryptionkey@0"
I20260619 16:41:37.716310 7074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:37.734432 7074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:37.736868 7074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:37.742518 7074 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43141
I20260619 16:41:37.742571 7328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43141 every 8 connection(s)
I20260619 16:41:37.744230 7074 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/master-2/data/info.pb
I20260619 16:41:37.746469 7329 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:41:37.747695 7329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4: Bootstrap starting.
W20260619 16:41:37.753707 7246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:37.754098 7246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:37.754173 7246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 16:41:37.756763 7329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:41:37.757107 7329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4: Bootstrap complete.
I20260619 16:41:37.757761 7329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: true } }
I20260619 16:41:37.757983 7329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b17354865b2491fb270179636349ab4, State: Initialized, Role: LEARNER
I20260619 16:41:37.758116 7329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: true } }
I20260619 16:41:37.758517 7335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: true } } }
I20260619 16:41:37.758710 7335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [sys.catalog]: This master's current role is: LEARNER
I20260619 16:41:37.758610 7329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:41:37.760874 7346 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260619 16:41:37.761288 7246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:37.761354 7246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:37.761457 7246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:41:37.768582 7246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42149
--tserver_master_addrs=127.0.98.62:43729,127.0.98.61:46879,127.0.98.60:43141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:37.769814 7246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:37.771004 7246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:37.781966 7349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:37.781966 7351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:37.782087 7348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:37.782209 7246 server_base.cc:1034] running on GCE node
I20260619 16:41:37.782721 7246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:37.783445 7246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:37.784684 7246 hybrid_clock.cc:648] HybridClock initialized: now 1781887297784658 us; error 41 us; skew 500 ppm
Jun 19 16:41:37 dist-test-slave-03cd krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887297, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:37.787798 7246 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 16:41:37.788774 7246 webserver.cc:460] Webserver started at http://127.0.98.1:42149/ using document root <none> and password file <none>
I20260619 16:41:37.789368 7246 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:37.789486 7246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:37.794111 7246 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:41:37.796548 7358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:37.798107 7246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:41:37.798254 7246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "715aa0a6a1334d59a2680f0b8cef1c85"
format_stamp: "Formatted at 2026-06-19 16:41:36 on dist-test-slave-03cd"
server_key: "11402e67b3772e9be1d01d3f9b963491"
server_key_iv: "4442af73e6a46c27b6d6d3d522244f67"
server_key_version: "encryptionkey@0"
I20260619 16:41:37.798936 7246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:37.820901 7246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:37.824461 7246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:37.824685 7246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:37.825380 7246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:37.826330 7246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:37.826399 7246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:37.826470 7246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:37.826521 7246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:37.837957 7246 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41841
I20260619 16:41:37.837973 7471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41841 every 8 connection(s)
I20260619 16:41:37.839041 7246 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:41:37.841040 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 7246
I20260619 16:41:37.841223 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 6697
I20260619 16:41:37.851146 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41495
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43729
--webserver_port=40787
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43729,127.0.98.61:46879
--tserver_master_addrs=127.0.98.62:43729,127.0.98.61:46879,127.0.98.60:43141 with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:41:37 dist-test-slave-03cd krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887297, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 19 16:41:37 dist-test-slave-03cd krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887297, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 16:41:37 dist-test-slave-03cd krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887297, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 16:41:37.861308 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.841054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33235 (local address 127.0.98.62:43729)
0619 16:41:37.841441 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:37.841445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:37.843254 (+ 1809us) server_negotiation.cc:407] Connection header received
0619 16:41:37.844243 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:37.844249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:37.844317 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:37.844413 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:37.846377 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.847306 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.849970 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.850360 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.850456 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.855708 (+ 5252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:37.855713 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:37.855715 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:37.855744 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:37.858104 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.859097 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.859102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.859106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.859226 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.859785 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.859791 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.859796 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.860128 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:37.860241 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:37.860947 (+ 706us) server_negotiation.cc:299] Negotiation successful
0619 16:41:37.861134 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299}
I20260619 16:41:37.861313 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.843188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47855 (local address 127.0.98.61:46879)
0619 16:41:37.843300 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:37.843304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:37.843852 (+ 548us) server_negotiation.cc:407] Connection header received
0619 16:41:37.844153 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:37.844158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:37.844221 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:37.844271 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:37.846377 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.847192 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.849839 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.850332 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.850427 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.855724 (+ 5297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:37.855732 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:37.855738 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:37.855779 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:37.858442 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.859228 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.859232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.859233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.859316 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.859641 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.859645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.859647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.859931 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:37.860026 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:37.860927 (+ 901us) server_negotiation.cc:299] Negotiation successful
0619 16:41:37.861121 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:41:37.862944 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.841219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34897 (local address 127.0.98.60:43141)
0619 16:41:37.841689 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:37.841693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:37.842795 (+ 1102us) server_negotiation.cc:407] Connection header received
0619 16:41:37.844373 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:37.844379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:37.844436 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:37.844496 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:37.846377 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.847456 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.849839 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.850333 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.850427 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.855699 (+ 5272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:37.855705 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:37.855708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:37.855742 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:37.858104 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.859019 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.859023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.859025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.859116 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:37.859641 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:37.859645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:37.859647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:37.859931 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:37.860020 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:37.862630 (+ 2610us) server_negotiation.cc:299] Negotiation successful
0619 16:41:37.862772 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":81,"threads_started":1}
I20260619 16:41:37.863031 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.842837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46879 (local address 127.0.98.1:47855)
0619 16:41:37.843763 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:37.843777 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:37.843923 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:37.844608 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:37.844617 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:37.845340 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:37.846204 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.846222 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.847369 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.847372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.849653 (+ 2281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.849668 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.850673 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.850677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.851061 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.851996 (+ 935us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:37.852011 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:37.855394 (+ 3383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:37.858852 (+ 3458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.858855 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.858857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.859105 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.859429 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.859432 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.859434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.859532 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.860101 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:37.860108 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:37.860593 (+ 485us) client_negotiation.cc:769] Sending connection context
0619 16:41:37.861553 (+ 960us) client_negotiation.cc:240] Negotiation successful
0619 16:41:37.861925 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"mutex_wait_us":667,"spinlock_wait_cycles":37888,"thread_start_us":101,"threads_started":1}
I20260619 16:41:37.863076 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.841491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.98.1:33235)
0619 16:41:37.843155 (+ 1664us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:37.843173 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:37.844112 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:37.844608 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:37.844619 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:37.845343 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:37.846216 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.846230 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.847461 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.847467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.849798 (+ 2331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.849813 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.850846 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.850852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.851071 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.851955 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:37.851983 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:37.855466 (+ 3483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:37.858295 (+ 2829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.858298 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.858319 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.858912 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.859377 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.859383 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.859385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.859599 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.860351 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:37.860358 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:37.860601 (+ 243us) client_negotiation.cc:769] Sending connection context
0619 16:41:37.861509 (+ 908us) client_negotiation.cc:240] Negotiation successful
0619 16:41:37.862050 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1510,"mutex_wait_us":875,"spinlock_wait_cycles":33664,"thread_start_us":135,"threads_started":1}
I20260619 16:41:37.864347 7473 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43729
I20260619 16:41:37.864670 7473 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:37.865125 7478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:37.841491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43141 (local address 127.0.98.1:34897)
0619 16:41:37.842590 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:37.842689 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:37.843922 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:37.844620 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:37.844622 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:37.845359 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:37.846202 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.846222 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.847582 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.847585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.849653 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:37.849668 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:37.850673 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:37.850677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:37.851054 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:37.852522 (+ 1468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:37.852537 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:37.855394 (+ 2857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:37.858288 (+ 2894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.858296 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.858319 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.858886 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.859227 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:37.859230 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:37.859232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:37.859532 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:37.860127 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:37.860129 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:37.860612 (+ 483us) client_negotiation.cc:769] Sending connection context
0619 16:41:37.864815 (+ 4203us) client_negotiation.cc:240] Negotiation successful
0619 16:41:37.864994 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"mutex_wait_us":743,"spinlock_wait_cycles":19712,"thread_start_us":667,"threads_started":1}
I20260619 16:41:37.865404 7473 heartbeater.cc:511] Master 127.0.98.62:43729 requested a full tablet report, sending...
I20260619 16:41:37.865867 7472 heartbeater.cc:347] Connected to a master server at 127.0.98.60:43141
I20260619 16:41:37.865949 7472 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:37.866117 7472 heartbeater.cc:511] Master 127.0.98.60:43141 requested a full tablet report, sending...
I20260619 16:41:37.864362 7474 heartbeater.cc:347] Connected to a master server at 127.0.98.61:46879
I20260619 16:41:37.866263 7474 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:37.866468 7474 heartbeater.cc:511] Master 127.0.98.61:46879 requested a full tablet report, sending...
I20260619 16:41:37.867306 7293 ts_manager.cc:194] Registered new tserver with Master: 715aa0a6a1334d59a2680f0b8cef1c85 (127.0.98.1:41841)
I20260619 16:41:37.867368 6501 ts_manager.cc:194] Re-registered known tserver with Master: 715aa0a6a1334d59a2680f0b8cef1c85 (127.0.98.1:41841)
I20260619 16:41:37.868094 7102 ts_manager.cc:194] Registered new tserver with Master: 715aa0a6a1334d59a2680f0b8cef1c85 (127.0.98.1:41841)
I20260619 16:41:37.868160 6501 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33235
I20260619 16:41:37.890483 6960 heartbeater.cc:503] Master 127.0.98.62:43729 was elected leader, sending a full tablet report...
W20260619 16:41:37.988574 7480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:37.989109 7480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:37.989207 7480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:37.994437 7480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:37.994560 7480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:37.994670 7480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:41:38.002228 7480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41495
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40787
--tserver_master_addrs=127.0.98.62:43729,127.0.98.61:46879,127.0.98.60:43141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:38.004303 7480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:38.005839 7480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:38.018913 7486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:38.019012 7485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:38.019101 7480 server_base.cc:1034] running on GCE node
W20260619 16:41:38.019311 7488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:38.020007 7480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:38.020951 7480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:38.022192 7480 hybrid_clock.cc:648] HybridClock initialized: now 1781887298022142 us; error 65 us; skew 500 ppm
Jun 19 16:41:38 dist-test-slave-03cd krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887298, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:38.026331 7480 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 16:41:38.027424 7480 webserver.cc:460] Webserver started at http://127.0.98.2:40787/ using document root <none> and password file <none>
I20260619 16:41:38.028117 7480 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:38.028213 7480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:38.033423 7480 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:41:38.036602 7495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:38.039295 7480 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:41:38.039515 7480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "da90af8c16574881832a55a5861d2c8f"
format_stamp: "Formatted at 2026-06-19 16:41:36 on dist-test-slave-03cd"
server_key: "bc7fdef778629ae2f9a299aaa4d24c65"
server_key_iv: "3d40d03c9cf986dcfac787c09963b357"
server_key_version: "encryptionkey@0"
I20260619 16:41:38.040520 7480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:38.049054 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.038266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43141 (local address 127.0.0.1:36508)
0619 16:41:38.038500 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:38.038518 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:38.038685 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:38.039071 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:38.039075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:38.039356 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:38.039515 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:38.039524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.040762 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.040770 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:38.041815 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:38.041838 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.042709 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.042713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:38.042810 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.043802 (+ 992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:38.043824 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:38.044340 (+ 516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:38.046845 (+ 2505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:38.046852 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:38.046857 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:38.047181 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:38.047769 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:38.047773 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:38.047775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:38.047885 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:38.048471 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:38.048474 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:38.048595 (+ 121us) client_negotiation.cc:769] Sending connection context
0619 16:41:38.048708 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 16:41:38.048878 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260619 16:41:38.049342 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.038325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36508 (local address 127.0.98.60:43141)
0619 16:41:38.038504 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:38.038512 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:38.038601 (+ 89us) server_negotiation.cc:407] Connection header received
0619 16:41:38.038807 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:38.038816 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:38.038875 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:38.039070 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:38.039654 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.040506 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.042125 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.042533 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.042671 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.044620 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:38.044625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:38.044628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:38.044676 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:38.046672 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:38.047419 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:38.047423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:38.047425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:38.047567 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:38.048054 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:38.048057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:38.048059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:38.048349 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:38.048459 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:38.048822 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 16:41:38.049060 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:41:38.050071 7303 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 16:41:38.050616 7335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d328d7aca0674111beef001a9009b0f8. Latest consensus state: current_term: 1 leader_uuid: "d328d7aca0674111beef001a9009b0f8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: true } } }
I20260619 16:41:38.050805 7335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [sys.catalog]: This master's current role is: LEARNER
I20260619 16:41:38.066256 7480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:38.070713 7480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:38.070962 7480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:38.071717 7480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:38.072705 7480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:38.072758 7480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:38.072850 7480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:38.072887 7480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:38.083213 7480 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41495
I20260619 16:41:38.083233 7608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41495 every 8 connection(s)
I20260619 16:41:38.084471 7480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:41:38.091228 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 7480
I20260619 16:41:38.091454 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 6831
Jun 19 16:41:38 dist-test-slave-03cd krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887298, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 16:41:38 dist-test-slave-03cd krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887298, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 16:41:38.101181 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34813
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43729
--webserver_port=45809
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43729,127.0.98.61:46879
--tserver_master_addrs=127.0.98.62:43729,127.0.98.61:46879,127.0.98.60:43141 with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:41:38 dist-test-slave-03cd krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887298, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 16:41:38.107692 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.087147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34311 (local address 127.0.98.61:46879)
0619 16:41:38.087380 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:38.087384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:38.089012 (+ 1628us) server_negotiation.cc:407] Connection header received
0619 16:41:38.090227 (+ 1215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:38.090231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:38.090293 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:38.090356 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:38.092654 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.093502 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.095878 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.096231 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.096316 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.102146 (+ 5830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:38.102154 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:38.102161 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:38.102203 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:38.104790 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:38.105802 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:38.105810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:38.105815 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:38.105924 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:38.106285 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:38.106292 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:38.106297 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:38.106631 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:38.106740 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:38.107205 (+ 465us) server_negotiation.cc:299] Negotiation successful
0619 16:41:38.107373 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260619 16:41:38.108145 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.087029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49385 (local address 127.0.98.60:43141)
0619 16:41:38.087216 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:38.087222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:38.088957 (+ 1735us) server_negotiation.cc:407] Connection header received
0619 16:41:38.090243 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:38.090247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:38.090332 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:38.090419 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:38.094367 (+ 3948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.095345 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.096830 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.097350 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.097495 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.104052 (+ 6557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:38.104060 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:38.104064 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:38.104107 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:38.106078 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:38.106576 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:38.106580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:38.106582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:38.106676 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:38.106964 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:38.106969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:38.106974 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:38.107209 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:38.107291 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:38.107620 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 16:41:38.107953 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 16:41:38.108770 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.087007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59469 (local address 127.0.98.62:43729)
0619 16:41:38.087288 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:38.087294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:38.088695 (+ 1401us) server_negotiation.cc:407] Connection header received
0619 16:41:38.090227 (+ 1532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:38.090230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:38.090322 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:38.090390 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:38.092654 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.093502 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.095878 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.096231 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.096315 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.101153 (+ 4838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:38.101159 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:38.101163 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:38.101208 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:38.103932 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:38.104873 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:38.104877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:38.104879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:38.104985 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:38.105761 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:38.105768 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:38.105771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:38.106075 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:38.106180 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:38.108446 (+ 2266us) server_negotiation.cc:299] Negotiation successful
0619 16:41:38.108609 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260619 16:41:38.109167 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.087536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.98.2:59469)
0619 16:41:38.088499 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:38.088565 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:38.089987 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:38.090734 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:38.090742 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:38.091595 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:38.092455 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:38.092477 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.093677 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.093680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:38.095704 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:38.095720 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.096556 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.096560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:38.096909 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.097947 (+ 1038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:38.097981 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:38.100825 (+ 2844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:38.104109 (+ 3284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:38.104118 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:38.104150 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:38.104661 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:38.105104 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:38.105107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:38.105110 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:38.105635 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:38.106252 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:38.106265 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:38.106763 (+ 498us) client_negotiation.cc:769] Sending connection context
0619 16:41:38.107008 (+ 245us) client_negotiation.cc:240] Negotiation successful
0619 16:41:38.107488 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":342,"spinlock_wait_cycles":768,"thread_start_us":707,"threads_started":1}
I20260619 16:41:38.109236 7613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.087538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46879 (local address 127.0.98.2:34311)
0619 16:41:38.088915 (+ 1377us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:38.088931 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:38.089987 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:38.090734 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:38.090742 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:38.092054 (+ 1312us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:38.092455 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:38.092477 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.093677 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.093680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:38.095704 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:38.095720 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.096556 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.096560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:38.096909 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.098219 (+ 1310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:38.098583 (+ 364us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:38.101904 (+ 3321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:38.104962 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:38.104970 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:38.104975 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:38.105655 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:38.106053 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:38.106059 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:38.106064 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:38.106171 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:38.106818 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:38.106821 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:38.106966 (+ 145us) client_negotiation.cc:769] Sending connection context
0619 16:41:38.107898 (+ 932us) client_negotiation.cc:240] Negotiation successful
0619 16:41:38.108100 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1263,"mutex_wait_us":596,"spinlock_wait_cycles":99584,"thread_start_us":214,"threads_started":1}
I20260619 16:41:38.109884 7615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.087537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43141 (local address 127.0.98.2:49385)
0619 16:41:38.088804 (+ 1267us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:38.088825 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:38.089951 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:38.090731 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:38.090740 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:38.093890 (+ 3150us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:38.094129 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:38.094143 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.095544 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.095550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:38.096661 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:38.096676 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.097781 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.097787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:38.097948 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.098621 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:38.098643 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:38.103762 (+ 5119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:38.106212 (+ 2450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:38.106216 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:38.106219 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:38.106454 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:38.106784 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:38.106787 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:38.106788 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:38.106864 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:38.107347 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:38.107350 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:38.107432 (+ 82us) client_negotiation.cc:769] Sending connection context
0619 16:41:38.108341 (+ 909us) client_negotiation.cc:240] Negotiation successful
0619 16:41:38.108501 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1145,"mutex_wait_us":30,"spinlock_wait_cycles":9088,"thread_start_us":243,"threads_started":1}
I20260619 16:41:38.110924 7610 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43729
I20260619 16:41:38.110970 7609 heartbeater.cc:347] Connected to a master server at 127.0.98.60:43141
I20260619 16:41:38.111243 7609 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:38.111243 7610 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:38.111761 7611 heartbeater.cc:347] Connected to a master server at 127.0.98.61:46879
I20260619 16:41:38.111873 7611 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:38.112167 7609 heartbeater.cc:511] Master 127.0.98.60:43141 requested a full tablet report, sending...
I20260619 16:41:38.112208 7611 heartbeater.cc:511] Master 127.0.98.61:46879 requested a full tablet report, sending...
I20260619 16:41:38.112167 7610 heartbeater.cc:511] Master 127.0.98.62:43729 requested a full tablet report, sending...
I20260619 16:41:38.113667 6501 ts_manager.cc:194] Re-registered known tserver with Master: da90af8c16574881832a55a5861d2c8f (127.0.98.2:41495)
I20260619 16:41:38.113772 7293 ts_manager.cc:194] Registered new tserver with Master: da90af8c16574881832a55a5861d2c8f (127.0.98.2:41495)
I20260619 16:41:38.113772 7102 ts_manager.cc:194] Registered new tserver with Master: da90af8c16574881832a55a5861d2c8f (127.0.98.2:41495)
I20260619 16:41:38.114645 6501 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59469
W20260619 16:41:38.232993 7616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:38.233320 7616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:38.233395 7616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:38.237524 7616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:38.237629 7616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:38.237726 7616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:41:38.243692 7616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45809
--tserver_master_addrs=127.0.98.62:43729,127.0.98.61:46879,127.0.98.60:43141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:38.245092 7616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:38.246455 7616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:38.259105 7621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:38.259116 7622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:38.259497 7624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:38.259423 7616 server_base.cc:1034] running on GCE node
I20260619 16:41:38.260064 7616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:38.260852 7616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:38.262058 7616 hybrid_clock.cc:648] HybridClock initialized: now 1781887298262025 us; error 68 us; skew 500 ppm
Jun 19 16:41:38 dist-test-slave-03cd krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887298, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:38.265345 7616 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 16:41:38.266357 7616 webserver.cc:460] Webserver started at http://127.0.98.3:45809/ using document root <none> and password file <none>
I20260619 16:41:38.266922 7616 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:38.266997 7616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:38.271461 7616 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:41:38.274034 7631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:38.275818 7616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:41:38.275992 7616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "a22a1178390044a9a7cc957008be869f"
format_stamp: "Formatted at 2026-06-19 16:41:36 on dist-test-slave-03cd"
server_key: "7bb284ee0b3b82fe57353e7345dc4054"
server_key_iv: "43fd3cbf89e68985338474ed7ab23ff3"
server_key_version: "encryptionkey@0"
I20260619 16:41:38.276639 7616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:38.298852 7616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:38.302881 7616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:38.303067 7616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:38.303864 7616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:38.304904 7616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:38.304963 7616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:38.305009 7616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:38.305027 7616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:38.318877 7616 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34813
I20260619 16:41:38.319103 7744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34813 every 8 connection(s)
I20260619 16:41:38.320243 7616 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:41:38.323709 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 7616
I20260619 16:41:38.338762 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.326319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40332 (local address 127.0.98.62:43729)
0619 16:41:38.327329 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:38.327349 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:38.327370 (+ 21us) server_negotiation.cc:407] Connection header received
0619 16:41:38.327445 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:38.327448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:38.327519 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:38.327743 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:38.329143 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.330051 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.331278 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.331636 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.331767 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.333739 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:38.333744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:38.333748 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:38.333781 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:38.336480 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:38.337158 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:38.337161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:38.337163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:38.337270 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:38.337563 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:38.337566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:38.337568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:38.337770 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:38.337918 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:38.338255 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 16:41:38.338377 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":928,"thread_start_us":106,"threads_started":1}
I20260619 16:41:38.339278 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.326155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36520 (local address 127.0.98.60:43141)
0619 16:41:38.326372 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:38.326378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:38.327623 (+ 1245us) server_negotiation.cc:407] Connection header received
0619 16:41:38.328068 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:38.328080 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:38.328170 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:38.328253 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:38.329255 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.330049 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.332169 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.332862 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.332978 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.334844 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:38.334849 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:38.334852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:38.334884 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:38.336586 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:38.337590 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:38.337595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:38.337597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:38.337723 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:38.338061 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:38.338064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:38.338066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:38.338318 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:38.338467 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:38.338878 (+ 411us) server_negotiation.cc:299] Negotiation successful
0619 16:41:38.339048 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
Jun 19 16:41:38 dist-test-slave-03cd krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887298, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 16:41:38 dist-test-slave-03cd krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887298, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 16:41:38.348687 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.326934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40552 (local address 127.0.98.61:46879)
0619 16:41:38.330548 (+ 3614us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:38.330553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:38.331189 (+ 636us) server_negotiation.cc:407] Connection header received
0619 16:41:38.331250 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:38.331254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:38.331316 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:38.331405 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:38.332632 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.333650 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.335006 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.335426 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.335542 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.343409 (+ 7867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:38.343414 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:38.343417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:38.343449 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:38.345892 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:38.346493 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:38.346496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:38.346498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:38.346625 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:38.346957 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:38.346962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:38.346964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:38.347222 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:38.347372 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:38.348145 (+ 773us) server_negotiation.cc:299] Negotiation successful
0619 16:41:38.348492 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3524,"thread_start_us":114,"threads_started":1}
I20260619 16:41:38.346700 6501 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:41:38.352392 6501 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Jun 19 16:41:38 dist-test-slave-03cd krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887298, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 16:41:38.354378 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.325958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55983 (local address 127.0.98.62:43729)
0619 16:41:38.326239 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:38.326245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:38.330690 (+ 4445us) server_negotiation.cc:407] Connection header received
0619 16:41:38.330807 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:38.330811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:38.330870 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:38.330946 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:38.339083 (+ 8137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.339819 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.342142 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.342690 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.342888 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.349729 (+ 6841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:38.349734 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:38.349736 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:38.349768 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:38.351879 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:38.352522 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:38.352526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:38.352528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:38.352606 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:38.352944 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:38.352947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:38.352949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:38.353165 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:38.353236 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:38.354054 (+ 818us) server_negotiation.cc:299] Negotiation successful
0619 16:41:38.354199 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184}
I20260619 16:41:38.354378 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.323474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47637 (local address 127.0.98.61:46879)
0619 16:41:38.323628 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:38.323633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:38.336697 (+ 13064us) server_negotiation.cc:407] Connection header received
0619 16:41:38.336776 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:38.336778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:38.336852 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:38.336940 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:38.339741 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.340606 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.342173 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.342516 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.342631 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.349321 (+ 6690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:38.349326 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:38.349329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:38.349358 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:38.351417 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:38.352190 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:38.352196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:38.352201 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:38.352288 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:38.352733 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:38.352736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:38.352738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:38.353008 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:38.353089 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:38.354053 (+ 964us) server_negotiation.cc:299] Negotiation successful
0619 16:41:38.354199 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:41:38.356521 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.323209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46879 (local address 127.0.98.3:47637)
0619 16:41:38.336073 (+ 12864us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:38.336097 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:38.336243 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:38.337126 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:38.337133 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:38.339363 (+ 2230us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:38.339530 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:38.339539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.341026 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.341030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:38.342031 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:38.342045 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.342894 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.342898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:38.343249 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.346049 (+ 2800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:38.346062 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:38.349074 (+ 3012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:38.351576 (+ 2502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:38.351586 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:38.351610 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:38.352073 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:38.352413 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:38.352417 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:38.352419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:38.352625 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:38.353156 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:38.353164 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:38.353732 (+ 568us) client_negotiation.cc:769] Sending connection context
0619 16:41:38.354625 (+ 893us) client_negotiation.cc:240] Negotiation successful
0619 16:41:38.355287 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12656,"mutex_wait_us":1477,"spinlock_wait_cycles":4736,"thread_start_us":93,"threads_started":1}
I20260619 16:41:38.356521 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.323210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.98.3:55983)
0619 16:41:38.330562 (+ 7352us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:38.330581 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:38.330700 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:38.332459 (+ 1759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:38.332477 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:38.338723 (+ 6246us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:38.338923 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:38.338938 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.340422 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.340426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:38.341995 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:38.342008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.343928 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.343934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:38.344055 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.345751 (+ 1696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:38.345781 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:38.349548 (+ 3767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:38.352013 (+ 2465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:38.352019 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:38.352022 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:38.352414 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:38.352716 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:38.352720 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:38.352722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:38.352818 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:38.353287 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:38.353289 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:38.353732 (+ 443us) client_negotiation.cc:769] Sending connection context
0619 16:41:38.354592 (+ 860us) client_negotiation.cc:240] Negotiation successful
0619 16:41:38.355226 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7212,"mutex_wait_us":670,"spinlock_wait_cycles":14848,"thread_start_us":101,"threads_started":1}
I20260619 16:41:38.358430 7746 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43729
I20260619 16:41:38.358819 7746 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:38.358927 7747 heartbeater.cc:347] Connected to a master server at 127.0.98.61:46879
I20260619 16:41:38.359035 7747 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:38.359648 7747 heartbeater.cc:511] Master 127.0.98.61:46879 requested a full tablet report, sending...
I20260619 16:41:38.359619 7746 heartbeater.cc:511] Master 127.0.98.62:43729 requested a full tablet report, sending...
I20260619 16:41:38.360025 7255 sys_catalog.cc:455] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d328d7aca0674111beef001a9009b0f8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: true } } }
I20260619 16:41:38.360183 7255 sys_catalog.cc:458] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:38.360301 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.326717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57679 (local address 127.0.98.60:43141)
0619 16:41:38.336365 (+ 9648us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:38.336370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:38.336386 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:41:38.336429 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:38.336432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:38.336488 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:38.342217 (+ 5729us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:38.342334 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.343178 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.344523 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.344900 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.345023 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.354987 (+ 9964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:38.354993 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:38.354996 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:38.355037 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:38.357444 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:38.358084 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:38.358088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:38.358090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:38.358201 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:38.358541 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:38.358544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:38.358546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:38.358888 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:38.359789 (+ 901us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:38.359952 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 16:41:38.360130 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9541,"thread_start_us":79,"threads_started":1}
I20260619 16:41:38.359620 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.323209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43141 (local address 127.0.98.3:57679)
0619 16:41:38.328248 (+ 5039us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:38.328348 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:38.329931 (+ 1583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:38.336967 (+ 7036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:38.336971 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:38.337744 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:38.338756 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:38.338791 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.343331 (+ 4540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.343336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:38.344258 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:38.344275 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.345183 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.345187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:38.345285 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.352015 (+ 6730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:38.352035 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:38.354255 (+ 2220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:38.357636 (+ 3381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:38.357640 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:38.357643 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:38.357941 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:38.358309 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:38.358313 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:38.358315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:38.358414 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:38.359010 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:38.359014 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:38.359140 (+ 126us) client_negotiation.cc:769] Sending connection context
0619 16:41:38.359231 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 16:41:38.359421 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4701,"thread_start_us":169,"threads_started":1}
I20260619 16:41:38.361327 6500 ts_manager.cc:194] Re-registered known tserver with Master: a22a1178390044a9a7cc957008be869f (127.0.98.3:34813)
I20260619 16:41:38.361409 7102 ts_manager.cc:194] Registered new tserver with Master: a22a1178390044a9a7cc957008be869f (127.0.98.3:34813)
I20260619 16:41:38.362322 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55983
I20260619 16:41:38.362739 7745 heartbeater.cc:347] Connected to a master server at 127.0.98.60:43141
I20260619 16:41:38.362864 7745 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:38.363090 7745 heartbeater.cc:511] Master 127.0.98.60:43141 requested a full tablet report, sending...
I20260619 16:41:38.365249 7293 ts_manager.cc:194] Registered new tserver with Master: a22a1178390044a9a7cc957008be869f (127.0.98.3:34813)
I20260619 16:41:38.366171 7255 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8: attempting to promote NON_VOTER 3b17354865b2491fb270179636349ab4 to VOTER
I20260619 16:41:38.366571 7255 consensus_queue.cc:237] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: false } }
I20260619 16:41:38.367933 7303 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [term 1 LEARNER]: Refusing update from remote peer d328d7aca0674111beef001a9009b0f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 16:41:38.368376 7763 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 16:41:38.368552 7112 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [term 1 FOLLOWER]: Refusing update from remote peer d328d7aca0674111beef001a9009b0f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 16:41:38.368953 7763 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 16:41:38.370091 7255 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3b17354865b2491fb270179636349ab4 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: false } } }
I20260619 16:41:38.370711 7303 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3b17354865b2491fb270179636349ab4 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: false } } }
I20260619 16:41:38.372010 7255 sys_catalog.cc:455] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d328d7aca0674111beef001a9009b0f8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: false } } }
I20260619 16:41:38.372143 7255 sys_catalog.cc:458] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:38.372277 7255 sys_catalog.cc:455] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d328d7aca0674111beef001a9009b0f8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: false } } }
I20260619 16:41:38.372123 7111 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3b17354865b2491fb270179636349ab4 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: false } } }
I20260619 16:41:38.372340 7255 sys_catalog.cc:458] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:38.372442 7255 sys_catalog.cc:455] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d328d7aca0674111beef001a9009b0f8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: false } } }
I20260619 16:41:38.372507 7255 sys_catalog.cc:458] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:38.372709 7335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d328d7aca0674111beef001a9009b0f8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: false } } }
I20260619 16:41:38.372859 7335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:41:38.374043 7772 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d328d7aca0674111beef001a9009b0f8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: false } } }
I20260619 16:41:38.374223 7772 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:41:38.389410 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.382923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34813 (local address 127.0.0.1:53464)
0619 16:41:38.383107 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:38.383127 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:38.383236 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:38.384231 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:38.384236 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:38.384261 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:38.384409 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:38.384420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.385892 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.385895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:38.387190 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:38.387199 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.388884 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.388887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:38.388982 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.389015 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 16:41:38.389117 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 16:41:38.389232 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 16:41:38.389452 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.383219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53464 (local address 127.0.98.3:34813)
0619 16:41:38.383583 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:38.383599 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:38.383626 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:41:38.383757 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:38.383764 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:38.384025 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:38.384256 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:38.384675 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.385773 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.387547 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.388728 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.388864 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.389043 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:38.389147 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 16:41:38.389251 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":144,"threads_started":1}
I20260619 16:41:38.390566 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.383918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35786 (local address 127.0.98.2:41495)
0619 16:41:38.385362 (+ 1444us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:38.385372 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:38.385688 (+ 316us) server_negotiation.cc:407] Connection header received
0619 16:41:38.385757 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:38.385762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:38.385949 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:38.386102 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:38.386371 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.387431 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.388966 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.389841 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.389972 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.390105 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:38.390301 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 16:41:38.390404 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1347,"thread_start_us":122,"threads_started":1}
I20260619 16:41:38.390631 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.383958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41495 (local address 127.0.0.1:35786)
0619 16:41:38.385503 (+ 1545us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:38.385517 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:38.385608 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:38.386088 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:38.386091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:38.386110 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:38.386230 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:38.386237 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.387558 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.387562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:38.388815 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:38.388827 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.389975 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.389979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:38.390082 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.390168 (+ 86us) client_negotiation.cc:769] Sending connection context
0619 16:41:38.390252 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 16:41:38.390439 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1487,"mutex_wait_us":87,"thread_start_us":108,"threads_started":1}
I20260619 16:41:38.390962 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.383264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41841 (local address 127.0.0.1:45320)
0619 16:41:38.384211 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:38.384226 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:38.384316 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:38.385143 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:38.385146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:38.385167 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:38.385457 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:38.385466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.387059 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.387062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:38.388224 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:38.388233 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.389493 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.389499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:38.390038 (+ 539us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.390165 (+ 127us) client_negotiation.cc:769] Sending connection context
0619 16:41:38.390252 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 16:41:38.390432 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":124,"thread_start_us":173,"threads_started":1}
I20260619 16:41:38.391121 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.383939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45320 (local address 127.0.98.1:41841)
0619 16:41:38.384601 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:38.384609 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:38.384650 (+ 41us) server_negotiation.cc:407] Connection header received
0619 16:41:38.384730 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:38.384735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:38.384928 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:38.385083 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:38.385638 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.386885 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.388366 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.389152 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.389258 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.389423 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:38.390818 (+ 1395us) server_negotiation.cc:299] Negotiation successful
0619 16:41:38.390979 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":105,"threads_started":1}
I20260619 16:41:38.393097 7679 tablet_service.cc:1467] Processing CreateTablet for tablet 46818a4631314dea8f082f4cacb8be6e (DEFAULT_TABLE table=test-table [id=a700bb78a1a54e58bf8131a9ea1228d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:38.394279 7679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46818a4631314dea8f082f4cacb8be6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:38.394403 7543 tablet_service.cc:1467] Processing CreateTablet for tablet 46818a4631314dea8f082f4cacb8be6e (DEFAULT_TABLE table=test-table [id=a700bb78a1a54e58bf8131a9ea1228d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:38.394409 7400 tablet_service.cc:1467] Processing CreateTablet for tablet 46818a4631314dea8f082f4cacb8be6e (DEFAULT_TABLE table=test-table [id=a700bb78a1a54e58bf8131a9ea1228d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:38.395815 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46818a4631314dea8f082f4cacb8be6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:38.395815 7543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46818a4631314dea8f082f4cacb8be6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:38.403297 7778 tablet_bootstrap.cc:492] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f: Bootstrap starting.
I20260619 16:41:38.405514 7778 tablet_bootstrap.cc:654] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:38.405845 7779 tablet_bootstrap.cc:492] T 46818a4631314dea8f082f4cacb8be6e P 715aa0a6a1334d59a2680f0b8cef1c85: Bootstrap starting.
I20260619 16:41:38.406440 7778 log.cc:824] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:38.407258 7780 tablet_bootstrap.cc:492] T 46818a4631314dea8f082f4cacb8be6e P da90af8c16574881832a55a5861d2c8f: Bootstrap starting.
I20260619 16:41:38.409230 7778 tablet_bootstrap.cc:492] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f: No bootstrap required, opened a new log
I20260619 16:41:38.409518 7778 ts_tablet_manager.cc:1397] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260619 16:41:38.409794 7779 tablet_bootstrap.cc:654] T 46818a4631314dea8f082f4cacb8be6e P 715aa0a6a1334d59a2680f0b8cef1c85: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:38.409793 7780 tablet_bootstrap.cc:654] T 46818a4631314dea8f082f4cacb8be6e P da90af8c16574881832a55a5861d2c8f: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:38.410712 7779 log.cc:824] T 46818a4631314dea8f082f4cacb8be6e P 715aa0a6a1334d59a2680f0b8cef1c85: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:38.410712 7780 log.cc:824] T 46818a4631314dea8f082f4cacb8be6e P da90af8c16574881832a55a5861d2c8f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:38.413374 7779 tablet_bootstrap.cc:492] T 46818a4631314dea8f082f4cacb8be6e P 715aa0a6a1334d59a2680f0b8cef1c85: No bootstrap required, opened a new log
I20260619 16:41:38.413441 7780 tablet_bootstrap.cc:492] T 46818a4631314dea8f082f4cacb8be6e P da90af8c16574881832a55a5861d2c8f: No bootstrap required, opened a new log
I20260619 16:41:38.413599 7780 ts_tablet_manager.cc:1397] T 46818a4631314dea8f082f4cacb8be6e P da90af8c16574881832a55a5861d2c8f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 16:41:38.413600 7779 ts_tablet_manager.cc:1397] T 46818a4631314dea8f082f4cacb8be6e P 715aa0a6a1334d59a2680f0b8cef1c85: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260619 16:41:38.414575 7778 raft_consensus.cc:348] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22a1178390044a9a7cc957008be869f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34813 } } peers { permanent_uuid: "715aa0a6a1334d59a2680f0b8cef1c85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41841 } } peers { permanent_uuid: "da90af8c16574881832a55a5861d2c8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41495 } }
I20260619 16:41:38.414966 7778 raft_consensus.cc:374] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:38.415023 7778 raft_consensus.cc:729] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22a1178390044a9a7cc957008be869f, State: Initialized, Role: FOLLOWER
I20260619 16:41:38.415625 7778 consensus_queue.cc:260] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22a1178390044a9a7cc957008be869f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34813 } } peers { permanent_uuid: "715aa0a6a1334d59a2680f0b8cef1c85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41841 } } peers { permanent_uuid: "da90af8c16574881832a55a5861d2c8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41495 } }
I20260619 16:41:38.417074 7778 ts_tablet_manager.cc:1428] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 16:41:38.417568 7746 heartbeater.cc:503] Master 127.0.98.62:43729 was elected leader, sending a full tablet report...
I20260619 16:41:38.419288 7779 raft_consensus.cc:348] T 46818a4631314dea8f082f4cacb8be6e P 715aa0a6a1334d59a2680f0b8cef1c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22a1178390044a9a7cc957008be869f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34813 } } peers { permanent_uuid: "715aa0a6a1334d59a2680f0b8cef1c85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41841 } } peers { permanent_uuid: "da90af8c16574881832a55a5861d2c8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41495 } }
I20260619 16:41:38.419651 7779 raft_consensus.cc:374] T 46818a4631314dea8f082f4cacb8be6e P 715aa0a6a1334d59a2680f0b8cef1c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:38.419744 7779 raft_consensus.cc:729] T 46818a4631314dea8f082f4cacb8be6e P 715aa0a6a1334d59a2680f0b8cef1c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 715aa0a6a1334d59a2680f0b8cef1c85, State: Initialized, Role: FOLLOWER
I20260619 16:41:38.420303 7779 consensus_queue.cc:260] T 46818a4631314dea8f082f4cacb8be6e P 715aa0a6a1334d59a2680f0b8cef1c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22a1178390044a9a7cc957008be869f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34813 } } peers { permanent_uuid: "715aa0a6a1334d59a2680f0b8cef1c85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41841 } } peers { permanent_uuid: "da90af8c16574881832a55a5861d2c8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41495 } }
I20260619 16:41:38.421149 7473 heartbeater.cc:503] Master 127.0.98.62:43729 was elected leader, sending a full tablet report...
I20260619 16:41:38.422763 7779 ts_tablet_manager.cc:1428] T 46818a4631314dea8f082f4cacb8be6e P 715aa0a6a1334d59a2680f0b8cef1c85: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260619 16:41:38.423409 7780 raft_consensus.cc:348] T 46818a4631314dea8f082f4cacb8be6e P da90af8c16574881832a55a5861d2c8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22a1178390044a9a7cc957008be869f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34813 } } peers { permanent_uuid: "715aa0a6a1334d59a2680f0b8cef1c85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41841 } } peers { permanent_uuid: "da90af8c16574881832a55a5861d2c8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41495 } }
I20260619 16:41:38.423760 7780 raft_consensus.cc:374] T 46818a4631314dea8f082f4cacb8be6e P da90af8c16574881832a55a5861d2c8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:38.423825 7780 raft_consensus.cc:729] T 46818a4631314dea8f082f4cacb8be6e P da90af8c16574881832a55a5861d2c8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da90af8c16574881832a55a5861d2c8f, State: Initialized, Role: FOLLOWER
I20260619 16:41:38.425113 7780 consensus_queue.cc:260] T 46818a4631314dea8f082f4cacb8be6e P da90af8c16574881832a55a5861d2c8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22a1178390044a9a7cc957008be869f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34813 } } peers { permanent_uuid: "715aa0a6a1334d59a2680f0b8cef1c85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41841 } } peers { permanent_uuid: "da90af8c16574881832a55a5861d2c8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41495 } }
I20260619 16:41:38.426043 7610 heartbeater.cc:503] Master 127.0.98.62:43729 was elected leader, sending a full tablet report...
I20260619 16:41:38.426254 7780 ts_tablet_manager.cc:1428] T 46818a4631314dea8f082f4cacb8be6e P da90af8c16574881832a55a5861d2c8f: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20260619 16:41:38.521279 7784 raft_consensus.cc:482] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:41:38.521566 7784 raft_consensus.cc:504] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22a1178390044a9a7cc957008be869f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34813 } } peers { permanent_uuid: "715aa0a6a1334d59a2680f0b8cef1c85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41841 } } peers { permanent_uuid: "da90af8c16574881832a55a5861d2c8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41495 } }
I20260619 16:41:38.522764 7784 leader_election.cc:288] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715aa0a6a1334d59a2680f0b8cef1c85 (127.0.98.1:41841), da90af8c16574881832a55a5861d2c8f (127.0.98.2:41495)
I20260619 16:41:38.529310 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.523526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44483 (local address 127.0.98.1:41841)
0619 16:41:38.523782 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:38.523788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:38.523877 (+ 89us) server_negotiation.cc:407] Connection header received
0619 16:41:38.524112 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:38.524119 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:38.524191 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:38.524334 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:38.524934 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.525722 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.527746 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.528382 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.528522 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.528600 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:38.529052 (+ 452us) server_negotiation.cc:299] Negotiation successful
0619 16:41:38.529143 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 16:41:38.529347 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.523891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38309 (local address 127.0.98.2:41495)
0619 16:41:38.524046 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:38.524050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:38.524068 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:41:38.524276 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:38.524279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:38.524336 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:38.524410 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:38.524923 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.525734 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.527746 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.528393 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.528529 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.528600 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:38.529052 (+ 452us) server_negotiation.cc:299] Negotiation successful
0619 16:41:38.529143 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:41:38.529389 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.523682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41495 (local address 127.0.98.3:38309)
0619 16:41:38.523917 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:38.523938 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:38.524051 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:38.524453 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:38.524455 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:38.524467 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:38.524696 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:38.524702 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.525911 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.525915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:38.527546 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:38.527564 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.528596 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.528601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:38.528813 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.528875 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 16:41:38.528979 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 16:41:38.529153 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":269}
I20260619 16:41:38.529389 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.523295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41841 (local address 127.0.98.3:44483)
0619 16:41:38.523722 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:38.523752 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:38.523934 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:38.524361 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:38.524365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:38.524388 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:38.524675 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:38.524686 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.525911 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.525915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:38.527546 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:38.527564 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.528612 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.528615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:38.528830 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.528870 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 16:41:38.528979 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 16:41:38.529153 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":243}
I20260619 16:41:38.530459 7563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46818a4631314dea8f082f4cacb8be6e" candidate_uuid: "a22a1178390044a9a7cc957008be869f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da90af8c16574881832a55a5861d2c8f" is_pre_election: true
I20260619 16:41:38.530579 7426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46818a4631314dea8f082f4cacb8be6e" candidate_uuid: "a22a1178390044a9a7cc957008be869f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "715aa0a6a1334d59a2680f0b8cef1c85" is_pre_election: true
I20260619 16:41:38.530917 7563 raft_consensus.cc:2454] T 46818a4631314dea8f082f4cacb8be6e P da90af8c16574881832a55a5861d2c8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22a1178390044a9a7cc957008be869f in term 0.
I20260619 16:41:38.530964 7426 raft_consensus.cc:2454] T 46818a4631314dea8f082f4cacb8be6e P 715aa0a6a1334d59a2680f0b8cef1c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22a1178390044a9a7cc957008be869f in term 0.
I20260619 16:41:38.531534 7634 leader_election.cc:302] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a22a1178390044a9a7cc957008be869f, da90af8c16574881832a55a5861d2c8f; no voters:
I20260619 16:41:38.531883 7784 raft_consensus.cc:2781] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:41:38.531984 7784 raft_consensus.cc:482] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:41:38.532030 7784 raft_consensus.cc:3037] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:38.533149 7784 raft_consensus.cc:504] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22a1178390044a9a7cc957008be869f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34813 } } peers { permanent_uuid: "715aa0a6a1334d59a2680f0b8cef1c85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41841 } } peers { permanent_uuid: "da90af8c16574881832a55a5861d2c8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41495 } }
I20260619 16:41:38.533525 7784 leader_election.cc:288] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f [CANDIDATE]: Term 1 election: Requested vote from peers 715aa0a6a1334d59a2680f0b8cef1c85 (127.0.98.1:41841), da90af8c16574881832a55a5861d2c8f (127.0.98.2:41495)
I20260619 16:41:38.534106 7426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46818a4631314dea8f082f4cacb8be6e" candidate_uuid: "a22a1178390044a9a7cc957008be869f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "715aa0a6a1334d59a2680f0b8cef1c85"
I20260619 16:41:38.534106 7563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46818a4631314dea8f082f4cacb8be6e" candidate_uuid: "a22a1178390044a9a7cc957008be869f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da90af8c16574881832a55a5861d2c8f"
I20260619 16:41:38.534253 7426 raft_consensus.cc:3037] T 46818a4631314dea8f082f4cacb8be6e P 715aa0a6a1334d59a2680f0b8cef1c85 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:38.534255 7563 raft_consensus.cc:3037] T 46818a4631314dea8f082f4cacb8be6e P da90af8c16574881832a55a5861d2c8f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:38.535637 7426 raft_consensus.cc:2454] T 46818a4631314dea8f082f4cacb8be6e P 715aa0a6a1334d59a2680f0b8cef1c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22a1178390044a9a7cc957008be869f in term 1.
I20260619 16:41:38.535637 7563 raft_consensus.cc:2454] T 46818a4631314dea8f082f4cacb8be6e P da90af8c16574881832a55a5861d2c8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22a1178390044a9a7cc957008be869f in term 1.
I20260619 16:41:38.536099 7634 leader_election.cc:302] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 715aa0a6a1334d59a2680f0b8cef1c85, a22a1178390044a9a7cc957008be869f; no voters:
I20260619 16:41:38.536280 7784 raft_consensus.cc:2781] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:38.536545 7784 raft_consensus.cc:686] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f [term 1 LEADER]: Becoming Leader. State: Replica: a22a1178390044a9a7cc957008be869f, State: Running, Role: LEADER
I20260619 16:41:38.536964 7784 consensus_queue.cc:237] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22a1178390044a9a7cc957008be869f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34813 } } peers { permanent_uuid: "715aa0a6a1334d59a2680f0b8cef1c85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41841 } } peers { permanent_uuid: "da90af8c16574881832a55a5861d2c8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41495 } }
I20260619 16:41:38.540580 6500 catalog_manager.cc:5495] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f reported cstate change: term changed from 0 to 1, leader changed from <none> to a22a1178390044a9a7cc957008be869f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a22a1178390044a9a7cc957008be869f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22a1178390044a9a7cc957008be869f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "715aa0a6a1334d59a2680f0b8cef1c85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da90af8c16574881832a55a5861d2c8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41495 } health_report { overall_health: UNKNOWN } } }
W20260619 16:41:38.572973 7748 tablet.cc:2369] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:41:38.586942 7612 tablet.cc:2369] T 46818a4631314dea8f082f4cacb8be6e P da90af8c16574881832a55a5861d2c8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:41:38.589516 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.585226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53474 (local address 127.0.98.3:34813)
0619 16:41:38.585390 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:38.585394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:38.585413 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:41:38.585505 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:38.585508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:38.585567 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:38.585698 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:41:38.585979 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.586557 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.587696 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.587971 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.588096 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.588470 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:41:38.589057 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:41:38.589215 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:38.589253 (+ 38us) server_negotiation.cc:299] Negotiation successful
0619 16:41:38.589330 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 16:41:38.591019 7475 tablet.cc:2369] T 46818a4631314dea8f082f4cacb8be6e P 715aa0a6a1334d59a2680f0b8cef1c85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:41:38.598131 7563 raft_consensus.cc:1261] T 46818a4631314dea8f082f4cacb8be6e P da90af8c16574881832a55a5861d2c8f [term 1 FOLLOWER]: Refusing update from remote peer a22a1178390044a9a7cc957008be869f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:38.598150 7426 raft_consensus.cc:1261] T 46818a4631314dea8f082f4cacb8be6e P 715aa0a6a1334d59a2680f0b8cef1c85 [term 1 FOLLOWER]: Refusing update from remote peer a22a1178390044a9a7cc957008be869f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:38.599239 7784 consensus_queue.cc:1035] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f [LEADER]: Connected to new peer: Peer: permanent_uuid: "715aa0a6a1334d59a2680f0b8cef1c85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:38.599509 7787 consensus_queue.cc:1035] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f [LEADER]: Connected to new peer: Peer: permanent_uuid: "da90af8c16574881832a55a5861d2c8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:38.608162 7790 mvcc.cc:205] Tried to move back new op lower bound from 7298610375046811648 to 7298610374809829376. Current Snapshot: MvccSnapshot[applied={T|T < 7298610375046811648}]
I20260619 16:41:38.619964 7792 mvcc.cc:205] Tried to move back new op lower bound from 7298610375046811648 to 7298610374809829376. Current Snapshot: MvccSnapshot[applied={T|T < 7298610375046811648}]
I20260619 16:41:38.625666 6501 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:41:38.625844 6501 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:41:38.631285 6501 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d328d7aca0674111beef001a9009b0f8: Sending DeleteTablet for 3 replicas of tablet 46818a4631314dea8f082f4cacb8be6e
I20260619 16:41:38.632373 7400 tablet_service.cc:1514] Processing DeleteTablet for tablet 46818a4631314dea8f082f4cacb8be6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45320
I20260619 16:41:38.632700 7679 tablet_service.cc:1514] Processing DeleteTablet for tablet 46818a4631314dea8f082f4cacb8be6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53464
I20260619 16:41:38.633114 7543 tablet_service.cc:1514] Processing DeleteTablet for tablet 46818a4631314dea8f082f4cacb8be6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35786
I20260619 16:41:38.633613 7803 tablet_replica.cc:330] T 46818a4631314dea8f082f4cacb8be6e P da90af8c16574881832a55a5861d2c8f: stopping tablet replica
I20260619 16:41:38.634048 7803 raft_consensus.cc:2229] T 46818a4631314dea8f082f4cacb8be6e P da90af8c16574881832a55a5861d2c8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:41:38.634461 7803 raft_consensus.cc:2258] T 46818a4631314dea8f082f4cacb8be6e P da90af8c16574881832a55a5861d2c8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:38.634900 7800 tablet_replica.cc:330] T 46818a4631314dea8f082f4cacb8be6e P 715aa0a6a1334d59a2680f0b8cef1c85: stopping tablet replica
I20260619 16:41:38.635532 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 6469
I20260619 16:41:38.637112 7800 raft_consensus.cc:2229] T 46818a4631314dea8f082f4cacb8be6e P 715aa0a6a1334d59a2680f0b8cef1c85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:41:38.637562 7800 raft_consensus.cc:2258] T 46818a4631314dea8f082f4cacb8be6e P 715aa0a6a1334d59a2680f0b8cef1c85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:38.638049 7801 tablet_replica.cc:330] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f: stopping tablet replica
I20260619 16:41:38.638393 7801 raft_consensus.cc:2229] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:41:38.638756 7803 ts_tablet_manager.cc:1905] T 46818a4631314dea8f082f4cacb8be6e P da90af8c16574881832a55a5861d2c8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:38.638856 7801 raft_consensus.cc:2258] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:38.640667 7801 ts_tablet_manager.cc:1905] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:38.642357 7803 ts_tablet_manager.cc:1918] T 46818a4631314dea8f082f4cacb8be6e P da90af8c16574881832a55a5861d2c8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:41:38.642506 7803 log.cc:1196] T 46818a4631314dea8f082f4cacb8be6e P da90af8c16574881832a55a5861d2c8f: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/wal/wals/46818a4631314dea8f082f4cacb8be6e
I20260619 16:41:38.642921 7803 ts_tablet_manager.cc:1939] T 46818a4631314dea8f082f4cacb8be6e P da90af8c16574881832a55a5861d2c8f: Deleting consensus metadata
I20260619 16:41:38.644261 7801 ts_tablet_manager.cc:1918] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:41:38.644382 7801 log.cc:1196] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/wal/wals/46818a4631314dea8f082f4cacb8be6e
W20260619 16:41:38.644481 7497 connection.cc:441] server connection from 127.0.0.1:35786 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35786 (request call id 1) could send its response
I20260619 16:41:38.644752 7801 ts_tablet_manager.cc:1939] T 46818a4631314dea8f082f4cacb8be6e P a22a1178390044a9a7cc957008be869f: Deleting consensus metadata
I20260619 16:41:38.645610 7800 ts_tablet_manager.cc:1905] T 46818a4631314dea8f082f4cacb8be6e P 715aa0a6a1334d59a2680f0b8cef1c85: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 16:41:38.648490 7632 connection.cc:441] server connection from 127.0.0.1:53464 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53464 (request call id 1) could send its response
I20260619 16:41:38.649567 7800 ts_tablet_manager.cc:1918] T 46818a4631314dea8f082f4cacb8be6e P 715aa0a6a1334d59a2680f0b8cef1c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:41:38.649742 7800 log.cc:1196] T 46818a4631314dea8f082f4cacb8be6e P 715aa0a6a1334d59a2680f0b8cef1c85: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-0/wal/wals/46818a4631314dea8f082f4cacb8be6e
I20260619 16:41:38.650233 7800 ts_tablet_manager.cc:1939] T 46818a4631314dea8f082f4cacb8be6e P 715aa0a6a1334d59a2680f0b8cef1c85: Deleting consensus metadata
W20260619 16:41:38.651736 7361 connection.cc:441] server connection from 127.0.0.1:45320 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45320 (request call id 1) could send its response
I20260619 16:41:38.653640 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.634273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40556 (local address 127.0.98.61:46879)
0619 16:41:38.634524 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:38.634538 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:38.634551 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:41:38.634592 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:38.634595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:38.634642 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:38.634734 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:38.635616 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.636443 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.646308 (+ 9865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.646653 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.646737 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.649344 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:38.649351 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:38.649358 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:38.649397 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:38.651329 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:38.652204 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:38.652209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:38.652211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:38.652308 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:38.652652 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:38.652655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:38.652657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:38.652853 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:38.652970 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:38.653319 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 16:41:38.653454 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185}
I20260619 16:41:38.660934 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.649280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40572 (local address 127.0.98.61:46879)
0619 16:41:38.649916 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:38.649922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:38.651088 (+ 1166us) server_negotiation.cc:407] Connection header received
0619 16:41:38.651310 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:38.651314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:38.651375 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:38.651431 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:38.652536 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.653339 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.654718 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.655034 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.655147 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.657058 (+ 1911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:38.657062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:38.657064 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:38.657090 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:38.658818 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:38.659359 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:38.659365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:38.659370 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:38.659458 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:38.659788 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:38.659791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:38.659793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:38.660022 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:38.660150 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:38.660586 (+ 436us) server_negotiation.cc:299] Negotiation successful
0619 16:41:38.660739 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522}
I20260619 16:41:38.661999 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.650465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36534 (local address 127.0.98.60:43141)
0619 16:41:38.650600 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:38.650605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:38.652430 (+ 1825us) server_negotiation.cc:407] Connection header received
0619 16:41:38.652608 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:38.652612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:38.652670 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:38.652725 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:38.653630 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.654477 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.655633 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.656013 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.656087 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.657962 (+ 1875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:38.657966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:38.657969 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:38.658003 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:38.660005 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:38.660565 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:38.660569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:38.660570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:38.660656 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:38.660964 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:38.660967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:38.660969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:38.661213 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:38.661318 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:38.661678 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 16:41:38.661817 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:41:38.665398 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:38.634379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36522 (local address 127.0.98.60:43141)
0619 16:41:38.634518 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:38.634523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:38.655156 (+ 20633us) server_negotiation.cc:407] Connection header received
0619 16:41:38.655348 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:38.655352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:38.655412 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:38.655468 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:38.657080 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.658208 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.659436 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:38.659837 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:38.659931 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:38.661723 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:38.661727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:38.661730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:38.661767 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:38.663395 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:38.663921 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:38.663924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:38.663926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:38.664010 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:38.664287 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:38.664290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:38.664292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:38.664510 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:38.664681 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:38.665029 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 16:41:38.665198 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:41:38.762614 7346 catalog_manager.cc:1259] Loaded cluster ID: 02c7ab1904fc44fdb1154a0c0059f3af
I20260619 16:41:38.762818 7346 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4: loading cluster ID for follower catalog manager: success
I20260619 16:41:38.764855 7346 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4: acquiring CA information for follower catalog manager: success
I20260619 16:41:38.766196 7346 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:41:39.615386 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:39.614851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.98.2:41959)
0619 16:41:39.615176 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:39.615284 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":102,"threads_started":1}
W20260619 16:41:39.615777 7610 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43729 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111)
I20260619 16:41:39.618120 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:39.617407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.98.1:47781)
0619 16:41:39.617820 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:39.617974 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":84,"threads_started":1}
I20260619 16:41:39.618413 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:39.617797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.98.3:42011)
0619 16:41:39.618041 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:39.618116 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":64,"threads_started":1}
W20260619 16:41:39.618517 7473 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43729 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111)
W20260619 16:41:39.618767 7746 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43729 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111)
I20260619 16:41:40.132174 7814 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d328d7aca0674111beef001a9009b0f8)
I20260619 16:41:40.132432 7815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d328d7aca0674111beef001a9009b0f8)
I20260619 16:41:40.132503 7814 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: false } }
I20260619 16:41:40.132625 7815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: false } }
I20260619 16:41:40.133236 7815 leader_election.cc:288] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d328d7aca0674111beef001a9009b0f8 (127.0.98.62:43729), fb0fc19974204bc5aeb5f9e3f065d834 (127.0.98.61:46879)
I20260619 16:41:40.134253 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.133539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.0.1:40798)
0619 16:41:40.133984 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:40.134101 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":78,"threads_started":1}
W20260619 16:41:40.134536 7278 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111)
I20260619 16:41:40.134866 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.134139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.0.1:40804)
0619 16:41:40.134577 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:40.134745 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":86,"threads_started":1}
W20260619 16:41:40.135205 7087 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111)
I20260619 16:41:40.135303 7814 leader_election.cc:288] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d328d7aca0674111beef001a9009b0f8 (127.0.98.62:43729), 3b17354865b2491fb270179636349ab4 (127.0.98.60:43141)
I20260619 16:41:40.136251 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.135892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.0.1:40824)
0619 16:41:40.136005 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:40.136066 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 16:41:40.136256 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.136062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.0.1:40812)
0619 16:41:40.136150 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:40.136194 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 16:41:40.136394 7087 leader_election.cc:334] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d328d7aca0674111beef001a9009b0f8 (127.0.98.62:43729): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111)
W20260619 16:41:40.136650 7278 leader_election.cc:334] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d328d7aca0674111beef001a9009b0f8 (127.0.98.62:43729): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111)
I20260619 16:41:40.138763 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.133526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46879 (local address 127.0.0.1:40588)
0619 16:41:40.133912 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:40.133932 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:40.134087 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:40.134480 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:40.134483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:40.134501 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:40.134641 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:40.134650 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:40.135822 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:40.135826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:40.137183 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:40.137193 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:40.138316 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:40.138320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:40.138406 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:40.138450 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 16:41:40.138528 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 16:41:40.138597 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":91,"threads_started":1}
I20260619 16:41:40.138826 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.133584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40588 (local address 127.0.98.61:46879)
0619 16:41:40.133913 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:40.133919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:40.134018 (+ 99us) server_negotiation.cc:407] Connection header received
0619 16:41:40.134290 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:40.134296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:40.134359 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:40.134758 (+ 399us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:40.134853 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:40.135609 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:40.137337 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:40.138154 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:40.138287 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:40.138361 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:40.138571 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 16:41:40.138660 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":243,"thread_start_us":82,"threads_started":1}
I20260619 16:41:40.139359 7111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b17354865b2491fb270179636349ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" is_pre_election: true
I20260619 16:41:40.139564 7111 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b17354865b2491fb270179636349ab4 in term 1.
I20260619 16:41:40.140048 7279 leader_election.cc:302] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b17354865b2491fb270179636349ab4, fb0fc19974204bc5aeb5f9e3f065d834; no voters: d328d7aca0674111beef001a9009b0f8
I20260619 16:41:40.140202 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.135011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36542 (local address 127.0.98.60:43141)
0619 16:41:40.135831 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:40.135841 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:40.135853 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:41:40.135888 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:40.135890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:40.135932 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:40.136428 (+ 496us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:40.136518 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:40.137210 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:40.138855 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:40.139443 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:40.139565 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:40.139732 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:40.139949 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 16:41:40.140073 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":622,"threads_started":1}
I20260619 16:41:40.140196 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.134639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43141 (local address 127.0.0.1:36542)
0619 16:41:40.135050 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:40.135069 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:40.135189 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:40.136068 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:40.136071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:40.136091 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:40.136193 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:40.136198 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:40.137337 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:40.137341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:40.138705 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:40.138713 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:40.139580 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:40.139584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:40.139734 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:40.139769 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 16:41:40.139898 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 16:41:40.140025 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":66,"thread_start_us":56,"threads_started":1}
I20260619 16:41:40.140295 7815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 16:41:40.140363 7815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d328d7aca0674111beef001a9009b0f8)
I20260619 16:41:40.140409 7815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:41:40.140926 7303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3b17354865b2491fb270179636349ab4" is_pre_election: true
I20260619 16:41:40.141882 7815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: false } }
I20260619 16:41:40.142220 7303 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fb0fc19974204bc5aeb5f9e3f065d834 in current term 2: Already voted for candidate 3b17354865b2491fb270179636349ab4 in this term.
I20260619 16:41:40.142426 7815 leader_election.cc:288] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [CANDIDATE]: Term 2 election: Requested vote from peers d328d7aca0674111beef001a9009b0f8 (127.0.98.62:43729), fb0fc19974204bc5aeb5f9e3f065d834 (127.0.98.61:46879)
I20260619 16:41:40.142899 7111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b17354865b2491fb270179636349ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fb0fc19974204bc5aeb5f9e3f065d834"
I20260619 16:41:40.143051 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.142734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.0.1:40834)
0619 16:41:40.142915 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:40.142983 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 16:41:40.143069 7111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:41:40.143106 7086 leader_election.cc:302] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb0fc19974204bc5aeb5f9e3f065d834; no voters: 3b17354865b2491fb270179636349ab4, d328d7aca0674111beef001a9009b0f8
I20260619 16:41:40.144136 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.143906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.0.1:40836)
0619 16:41:40.144021 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:40.144081 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 16:41:40.144251 7278 leader_election.cc:334] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d328d7aca0674111beef001a9009b0f8 (127.0.98.62:43729): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111)
I20260619 16:41:40.144685 7111 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b17354865b2491fb270179636349ab4 in term 2.
I20260619 16:41:40.144855 7814 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:41:40.145041 7279 leader_election.cc:302] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b17354865b2491fb270179636349ab4, fb0fc19974204bc5aeb5f9e3f065d834; no voters: d328d7aca0674111beef001a9009b0f8
I20260619 16:41:40.145243 7815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:41:40.145540 7815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [term 2 LEADER]: Becoming Leader. State: Replica: 3b17354865b2491fb270179636349ab4, State: Running, Role: LEADER
I20260619 16:41:40.145892 7815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: false } }
I20260619 16:41:40.147729 7824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b17354865b2491fb270179636349ab4. Latest consensus state: current_term: 2 leader_uuid: "3b17354865b2491fb270179636349ab4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: false } } }
I20260619 16:41:40.147933 7824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:40.148315 7826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:41:40.150761 7826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:41:40.151247 7826 catalog_manager.cc:1259] Loaded cluster ID: 02c7ab1904fc44fdb1154a0c0059f3af
I20260619 16:41:40.151297 7826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:41:40.152438 7826 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:41:40.152946 7826 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4: Loaded TSK: 0
I20260619 16:41:40.153885 7826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:41:40.205255 7292 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:41:40.208815 7292 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:41:40.212879 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.212582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.0.1:40860)
0619 16:41:40.212721 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:40.212790 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 16:41:40.213163 7111 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [term 2 FOLLOWER]: Refusing update from remote peer 3b17354865b2491fb270179636349ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260619 16:41:40.213891 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.213583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.0.1:40876)
0619 16:41:40.213720 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:40.213792 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 16:41:40.213963 7824 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260619 16:41:40.214237 7278 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 -> Peer d328d7aca0674111beef001a9009b0f8 (127.0.98.62:43729): Couldn't send request to peer d328d7aca0674111beef001a9009b0f8. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 16:41:40.216151 7814 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b17354865b2491fb270179636349ab4. Latest consensus state: current_term: 2 leader_uuid: "3b17354865b2491fb270179636349ab4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: false } } }
I20260619 16:41:40.216432 7814 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:41:40.217939 7815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "3b17354865b2491fb270179636349ab4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: false } } }
I20260619 16:41:40.218180 7815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:40.218530 7814 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "3b17354865b2491fb270179636349ab4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d328d7aca0674111beef001a9009b0f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43729 } } peers { permanent_uuid: "fb0fc19974204bc5aeb5f9e3f065d834" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46879 } attrs { promote: false } } peers { permanent_uuid: "3b17354865b2491fb270179636349ab4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43141 } attrs { promote: false } } }
I20260619 16:41:40.218704 7814 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0fc19974204bc5aeb5f9e3f065d834 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:41:40.231981 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.227480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53488 (local address 127.0.98.3:34813)
0619 16:41:40.227827 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:40.227833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:40.227849 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:41:40.227909 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:40.227912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:40.227974 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:40.228375 (+ 401us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:40.228482 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:40.229359 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:40.230707 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:40.231325 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:40.231438 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:40.231517 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:40.231748 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 16:41:40.231830 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":112,"threads_started":1}
I20260619 16:41:40.231984 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.227293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34813 (local address 127.0.0.1:53488)
0619 16:41:40.227432 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:40.227449 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:40.227531 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:40.228113 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:40.228116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:40.228136 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:40.228249 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:40.228255 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:40.229488 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:40.229491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:40.230539 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:40.230548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:40.231448 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:40.231452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:40.231553 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:40.231586 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 16:41:40.231700 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 16:41:40.231801 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 16:41:40.232653 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.227831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41495 (local address 127.0.0.1:35800)
0619 16:41:40.228219 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:40.228234 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:40.228319 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:40.228782 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:40.228787 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:40.228807 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:40.229163 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:40.229169 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:40.230214 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:40.230221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:40.231155 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:40.231164 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:40.232234 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:40.232238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:40.232346 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:40.232371 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 16:41:40.232443 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 16:41:40.232526 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":7,"thread_start_us":133,"threads_started":1}
I20260619 16:41:40.233111 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.227970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35800 (local address 127.0.98.2:41495)
0619 16:41:40.228530 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:40.228535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:40.228550 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:41:40.228599 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:40.228602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:40.228647 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:40.228751 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:40.229292 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:40.230095 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:40.231290 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:40.231966 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:40.232087 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:40.232166 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:40.232881 (+ 715us) server_negotiation.cc:299] Negotiation successful
0619 16:41:40.232982 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":151,"threads_started":1}
I20260619 16:41:40.233047 7679 tablet_service.cc:1467] Processing CreateTablet for tablet 92a7317794fa49768b3146e33630bfc9 (DEFAULT_TABLE table=test-table [id=f27ccc08f54b421c9ada18670235d196]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:40.233386 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.227544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41841 (local address 127.0.0.1:45328)
0619 16:41:40.227794 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:40.227812 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:40.227938 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:40.228816 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:40.228818 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:40.228829 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:40.228973 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:40.228980 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:40.230097 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:40.230101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:40.231304 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:40.231314 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:40.232910 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:40.232917 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:40.233020 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:40.233051 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 16:41:40.233119 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 16:41:40.233210 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":49}
I20260619 16:41:40.233400 7679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92a7317794fa49768b3146e33630bfc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:40.234108 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.228223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45328 (local address 127.0.98.1:41841)
0619 16:41:40.228537 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:40.228541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:40.228555 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:41:40.228618 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:40.228620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:40.228664 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:40.228870 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:40.229092 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:40.229952 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:40.231789 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:40.232665 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:40.233712 (+ 1047us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:40.233811 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:40.233854 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 16:41:40.233971 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":93,"threads_started":1}
I20260619 16:41:40.234622 7400 tablet_service.cc:1467] Processing CreateTablet for tablet 92a7317794fa49768b3146e33630bfc9 (DEFAULT_TABLE table=test-table [id=f27ccc08f54b421c9ada18670235d196]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:40.234933 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92a7317794fa49768b3146e33630bfc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:40.236542 7836 tablet_bootstrap.cc:492] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f: Bootstrap starting.
I20260619 16:41:40.236837 7543 tablet_service.cc:1467] Processing CreateTablet for tablet 92a7317794fa49768b3146e33630bfc9 (DEFAULT_TABLE table=test-table [id=f27ccc08f54b421c9ada18670235d196]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:40.237161 7543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92a7317794fa49768b3146e33630bfc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:40.237511 7837 tablet_bootstrap.cc:492] T 92a7317794fa49768b3146e33630bfc9 P 715aa0a6a1334d59a2680f0b8cef1c85: Bootstrap starting.
I20260619 16:41:40.237751 7836 tablet_bootstrap.cc:654] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:40.238525 7837 tablet_bootstrap.cc:654] T 92a7317794fa49768b3146e33630bfc9 P 715aa0a6a1334d59a2680f0b8cef1c85: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:40.239112 7836 tablet_bootstrap.cc:492] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f: No bootstrap required, opened a new log
I20260619 16:41:40.239226 7836 ts_tablet_manager.cc:1397] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:41:40.239714 7840 tablet_bootstrap.cc:492] T 92a7317794fa49768b3146e33630bfc9 P da90af8c16574881832a55a5861d2c8f: Bootstrap starting.
I20260619 16:41:40.239794 7836 raft_consensus.cc:348] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22a1178390044a9a7cc957008be869f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34813 } } peers { permanent_uuid: "715aa0a6a1334d59a2680f0b8cef1c85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41841 } } peers { permanent_uuid: "da90af8c16574881832a55a5861d2c8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41495 } }
I20260619 16:41:40.240127 7836 raft_consensus.cc:374] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:40.240203 7836 raft_consensus.cc:729] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22a1178390044a9a7cc957008be869f, State: Initialized, Role: FOLLOWER
I20260619 16:41:40.240408 7837 tablet_bootstrap.cc:492] T 92a7317794fa49768b3146e33630bfc9 P 715aa0a6a1334d59a2680f0b8cef1c85: No bootstrap required, opened a new log
I20260619 16:41:40.240428 7836 consensus_queue.cc:260] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22a1178390044a9a7cc957008be869f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34813 } } peers { permanent_uuid: "715aa0a6a1334d59a2680f0b8cef1c85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41841 } } peers { permanent_uuid: "da90af8c16574881832a55a5861d2c8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41495 } }
I20260619 16:41:40.240573 7840 tablet_bootstrap.cc:654] T 92a7317794fa49768b3146e33630bfc9 P da90af8c16574881832a55a5861d2c8f: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:40.240830 7837 ts_tablet_manager.cc:1397] T 92a7317794fa49768b3146e33630bfc9 P 715aa0a6a1334d59a2680f0b8cef1c85: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:41:40.241132 7836 ts_tablet_manager.cc:1428] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:41:40.241683 7837 raft_consensus.cc:348] T 92a7317794fa49768b3146e33630bfc9 P 715aa0a6a1334d59a2680f0b8cef1c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22a1178390044a9a7cc957008be869f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34813 } } peers { permanent_uuid: "715aa0a6a1334d59a2680f0b8cef1c85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41841 } } peers { permanent_uuid: "da90af8c16574881832a55a5861d2c8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41495 } }
I20260619 16:41:40.241910 7837 raft_consensus.cc:374] T 92a7317794fa49768b3146e33630bfc9 P 715aa0a6a1334d59a2680f0b8cef1c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:40.241997 7837 raft_consensus.cc:729] T 92a7317794fa49768b3146e33630bfc9 P 715aa0a6a1334d59a2680f0b8cef1c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 715aa0a6a1334d59a2680f0b8cef1c85, State: Initialized, Role: FOLLOWER
I20260619 16:41:40.242810 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.242162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.98.3:60165)
0619 16:41:40.242576 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:40.242677 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":225,"threads_started":1}
I20260619 16:41:40.242652 7837 consensus_queue.cc:260] T 92a7317794fa49768b3146e33630bfc9 P 715aa0a6a1334d59a2680f0b8cef1c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22a1178390044a9a7cc957008be869f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34813 } } peers { permanent_uuid: "715aa0a6a1334d59a2680f0b8cef1c85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41841 } } peers { permanent_uuid: "da90af8c16574881832a55a5861d2c8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41495 } }
I20260619 16:41:40.243330 7837 ts_tablet_manager.cc:1428] T 92a7317794fa49768b3146e33630bfc9 P 715aa0a6a1334d59a2680f0b8cef1c85: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:41:40.244807 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.244309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.98.1:45353)
0619 16:41:40.244598 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:40.244675 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":136,"threads_started":1}
I20260619 16:41:40.245617 7745 heartbeater.cc:503] Master 127.0.98.60:43141 was elected leader, sending a full tablet report...
I20260619 16:41:40.246032 7472 heartbeater.cc:503] Master 127.0.98.60:43141 was elected leader, sending a full tablet report...
I20260619 16:41:40.246637 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.246301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.98.3:54853)
0619 16:41:40.246473 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:40.246537 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 16:41:40.247674 7840 tablet_bootstrap.cc:492] T 92a7317794fa49768b3146e33630bfc9 P da90af8c16574881832a55a5861d2c8f: No bootstrap required, opened a new log
I20260619 16:41:40.247823 7840 ts_tablet_manager.cc:1397] T 92a7317794fa49768b3146e33630bfc9 P da90af8c16574881832a55a5861d2c8f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260619 16:41:40.248327 7840 raft_consensus.cc:348] T 92a7317794fa49768b3146e33630bfc9 P da90af8c16574881832a55a5861d2c8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22a1178390044a9a7cc957008be869f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34813 } } peers { permanent_uuid: "715aa0a6a1334d59a2680f0b8cef1c85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41841 } } peers { permanent_uuid: "da90af8c16574881832a55a5861d2c8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41495 } }
I20260619 16:41:40.248451 7840 raft_consensus.cc:374] T 92a7317794fa49768b3146e33630bfc9 P da90af8c16574881832a55a5861d2c8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:40.248476 7840 raft_consensus.cc:729] T 92a7317794fa49768b3146e33630bfc9 P da90af8c16574881832a55a5861d2c8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da90af8c16574881832a55a5861d2c8f, State: Initialized, Role: FOLLOWER
I20260619 16:41:40.248622 7840 consensus_queue.cc:260] T 92a7317794fa49768b3146e33630bfc9 P da90af8c16574881832a55a5861d2c8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22a1178390044a9a7cc957008be869f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34813 } } peers { permanent_uuid: "715aa0a6a1334d59a2680f0b8cef1c85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41841 } } peers { permanent_uuid: "da90af8c16574881832a55a5861d2c8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41495 } }
I20260619 16:41:40.248967 7840 ts_tablet_manager.cc:1428] T 92a7317794fa49768b3146e33630bfc9 P da90af8c16574881832a55a5861d2c8f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:41:40.250571 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.249506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.98.2:56177)
0619 16:41:40.250364 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:40.250452 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":158,"threads_started":1}
I20260619 16:41:40.250877 7609 heartbeater.cc:503] Master 127.0.98.60:43141 was elected leader, sending a full tablet report...
I20260619 16:41:40.251888 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.251570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.98.2:45357)
0619 16:41:40.251742 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:40.251810 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 16:41:40.334331 7841 raft_consensus.cc:482] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:41:40.334509 7841 raft_consensus.cc:504] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22a1178390044a9a7cc957008be869f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34813 } } peers { permanent_uuid: "715aa0a6a1334d59a2680f0b8cef1c85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41841 } } peers { permanent_uuid: "da90af8c16574881832a55a5861d2c8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41495 } }
I20260619 16:41:40.334932 7841 leader_election.cc:288] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 715aa0a6a1334d59a2680f0b8cef1c85 (127.0.98.1:41841), da90af8c16574881832a55a5861d2c8f (127.0.98.2:41495)
I20260619 16:41:40.335861 7563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92a7317794fa49768b3146e33630bfc9" candidate_uuid: "a22a1178390044a9a7cc957008be869f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da90af8c16574881832a55a5861d2c8f" is_pre_election: true
I20260619 16:41:40.336071 7563 raft_consensus.cc:2454] T 92a7317794fa49768b3146e33630bfc9 P da90af8c16574881832a55a5861d2c8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22a1178390044a9a7cc957008be869f in term 0.
I20260619 16:41:40.336122 7426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92a7317794fa49768b3146e33630bfc9" candidate_uuid: "a22a1178390044a9a7cc957008be869f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "715aa0a6a1334d59a2680f0b8cef1c85" is_pre_election: true
I20260619 16:41:40.336346 7426 raft_consensus.cc:2454] T 92a7317794fa49768b3146e33630bfc9 P 715aa0a6a1334d59a2680f0b8cef1c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22a1178390044a9a7cc957008be869f in term 0.
I20260619 16:41:40.336464 7634 leader_election.cc:302] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a22a1178390044a9a7cc957008be869f, da90af8c16574881832a55a5861d2c8f; no voters:
I20260619 16:41:40.336720 7841 raft_consensus.cc:2781] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:41:40.336803 7841 raft_consensus.cc:482] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:41:40.336851 7841 raft_consensus.cc:3037] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:40.338634 7841 raft_consensus.cc:504] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22a1178390044a9a7cc957008be869f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34813 } } peers { permanent_uuid: "715aa0a6a1334d59a2680f0b8cef1c85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41841 } } peers { permanent_uuid: "da90af8c16574881832a55a5861d2c8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41495 } }
I20260619 16:41:40.339355 7841 leader_election.cc:288] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f [CANDIDATE]: Term 1 election: Requested vote from peers 715aa0a6a1334d59a2680f0b8cef1c85 (127.0.98.1:41841), da90af8c16574881832a55a5861d2c8f (127.0.98.2:41495)
I20260619 16:41:40.340024 7563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92a7317794fa49768b3146e33630bfc9" candidate_uuid: "a22a1178390044a9a7cc957008be869f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da90af8c16574881832a55a5861d2c8f"
I20260619 16:41:40.340191 7563 raft_consensus.cc:3037] T 92a7317794fa49768b3146e33630bfc9 P da90af8c16574881832a55a5861d2c8f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:40.340111 7426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92a7317794fa49768b3146e33630bfc9" candidate_uuid: "a22a1178390044a9a7cc957008be869f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "715aa0a6a1334d59a2680f0b8cef1c85"
I20260619 16:41:40.340322 7426 raft_consensus.cc:3037] T 92a7317794fa49768b3146e33630bfc9 P 715aa0a6a1334d59a2680f0b8cef1c85 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:40.341351 7563 raft_consensus.cc:2454] T 92a7317794fa49768b3146e33630bfc9 P da90af8c16574881832a55a5861d2c8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22a1178390044a9a7cc957008be869f in term 1.
I20260619 16:41:40.341900 7426 raft_consensus.cc:2454] T 92a7317794fa49768b3146e33630bfc9 P 715aa0a6a1334d59a2680f0b8cef1c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22a1178390044a9a7cc957008be869f in term 1.
I20260619 16:41:40.342024 7634 leader_election.cc:302] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a22a1178390044a9a7cc957008be869f, da90af8c16574881832a55a5861d2c8f; no voters:
I20260619 16:41:40.342365 7841 raft_consensus.cc:2781] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:40.342690 7841 raft_consensus.cc:686] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f [term 1 LEADER]: Becoming Leader. State: Replica: a22a1178390044a9a7cc957008be869f, State: Running, Role: LEADER
I20260619 16:41:40.342891 7841 consensus_queue.cc:237] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22a1178390044a9a7cc957008be869f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34813 } } peers { permanent_uuid: "715aa0a6a1334d59a2680f0b8cef1c85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41841 } } peers { permanent_uuid: "da90af8c16574881832a55a5861d2c8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41495 } }
I20260619 16:41:40.343523 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.343267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.98.3:38515)
0619 16:41:40.343373 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:40.343442 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 16:41:40.345891 7292 catalog_manager.cc:5495] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f reported cstate change: term changed from 0 to 1, leader changed from <none> to a22a1178390044a9a7cc957008be869f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a22a1178390044a9a7cc957008be869f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22a1178390044a9a7cc957008be869f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "715aa0a6a1334d59a2680f0b8cef1c85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da90af8c16574881832a55a5861d2c8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41495 } health_report { overall_health: UNKNOWN } } }
I20260619 16:41:40.371277 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.367490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53492 (local address 127.0.98.3:34813)
0619 16:41:40.367727 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:40.367732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:40.367842 (+ 110us) server_negotiation.cc:407] Connection header received
0619 16:41:40.368155 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:40.368159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:40.368222 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:40.368419 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:41:40.368706 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:40.369238 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:40.370210 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:40.370427 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:40.370538 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:40.370787 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:41:40.370870 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:41:40.370951 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:40.371063 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 16:41:40.371119 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260619 16:41:40.374140 7563 raft_consensus.cc:1261] T 92a7317794fa49768b3146e33630bfc9 P da90af8c16574881832a55a5861d2c8f [term 1 FOLLOWER]: Refusing update from remote peer a22a1178390044a9a7cc957008be869f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:40.374126 7426 raft_consensus.cc:1261] T 92a7317794fa49768b3146e33630bfc9 P 715aa0a6a1334d59a2680f0b8cef1c85 [term 1 FOLLOWER]: Refusing update from remote peer a22a1178390044a9a7cc957008be869f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:40.374719 7848 consensus_queue.cc:1035] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f [LEADER]: Connected to new peer: Peer: permanent_uuid: "da90af8c16574881832a55a5861d2c8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:40.375046 7841 consensus_queue.cc:1035] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f [LEADER]: Connected to new peer: Peer: permanent_uuid: "715aa0a6a1334d59a2680f0b8cef1c85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:40.377686 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.377329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.98.1:33517)
0619 16:41:40.377488 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:40.377571 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 16:41:40.378326 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.377845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.98.2:39455)
0619 16:41:40.378144 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:40.378216 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218}
I20260619 16:41:40.383613 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.383210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.98.2:59959)
0619 16:41:40.383402 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:40.383477 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260619 16:41:40.386078 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.385731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.98.3:56423)
0619 16:41:40.385917 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:40.385984 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260619 16:41:40.386544 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.386116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43729 (local address 127.0.98.1:57263)
0619 16:41:40.386386 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:40.386455 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260619 16:41:40.386955 7293 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:41:40.387138 7293 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:41:40.393115 7293 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b17354865b2491fb270179636349ab4: Sending DeleteTablet for 3 replicas of tablet 92a7317794fa49768b3146e33630bfc9
I20260619 16:41:40.394289 7679 tablet_service.cc:1514] Processing DeleteTablet for tablet 92a7317794fa49768b3146e33630bfc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:40 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:53488
I20260619 16:41:40.394603 7400 tablet_service.cc:1514] Processing DeleteTablet for tablet 92a7317794fa49768b3146e33630bfc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:40 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:45328
I20260619 16:41:40.394601 7543 tablet_service.cc:1514] Processing DeleteTablet for tablet 92a7317794fa49768b3146e33630bfc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:40 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:35800
I20260619 16:41:40.395237 7863 tablet_replica.cc:330] T 92a7317794fa49768b3146e33630bfc9 P 715aa0a6a1334d59a2680f0b8cef1c85: stopping tablet replica
I20260619 16:41:40.395255 7861 tablet_replica.cc:330] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f: stopping tablet replica
I20260619 16:41:40.395421 7861 raft_consensus.cc:2229] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:41:40.395637 7861 raft_consensus.cc:2258] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:40.395597 7863 raft_consensus.cc:2229] T 92a7317794fa49768b3146e33630bfc9 P 715aa0a6a1334d59a2680f0b8cef1c85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:41:40.395886 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 7246
I20260619 16:41:40.395889 7863 raft_consensus.cc:2258] T 92a7317794fa49768b3146e33630bfc9 P 715aa0a6a1334d59a2680f0b8cef1c85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:40.396296 7862 tablet_replica.cc:330] T 92a7317794fa49768b3146e33630bfc9 P da90af8c16574881832a55a5861d2c8f: stopping tablet replica
I20260619 16:41:40.396581 7862 raft_consensus.cc:2229] T 92a7317794fa49768b3146e33630bfc9 P da90af8c16574881832a55a5861d2c8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:41:40.396770 7862 raft_consensus.cc:2258] T 92a7317794fa49768b3146e33630bfc9 P da90af8c16574881832a55a5861d2c8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:40.397255 7861 ts_tablet_manager.cc:1905] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:40.399449 7862 ts_tablet_manager.cc:1905] T 92a7317794fa49768b3146e33630bfc9 P da90af8c16574881832a55a5861d2c8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:40.400933 7861 ts_tablet_manager.cc:1918] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:41:40.401058 7861 log.cc:1196] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-2/wal/wals/92a7317794fa49768b3146e33630bfc9
I20260619 16:41:40.401376 7861 ts_tablet_manager.cc:1939] T 92a7317794fa49768b3146e33630bfc9 P a22a1178390044a9a7cc957008be869f: Deleting consensus metadata
I20260619 16:41:40.402227 7279 catalog_manager.cc:4841] TS a22a1178390044a9a7cc957008be869f (127.0.98.3:34813): tablet 92a7317794fa49768b3146e33630bfc9 (table test-table [id=f27ccc08f54b421c9ada18670235d196]) successfully deleted
I20260619 16:41:40.403610 7862 ts_tablet_manager.cc:1918] T 92a7317794fa49768b3146e33630bfc9 P da90af8c16574881832a55a5861d2c8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:41:40.403770 7862 log.cc:1196] T 92a7317794fa49768b3146e33630bfc9 P da90af8c16574881832a55a5861d2c8f: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887259574980-392-0/minicluster-data/ts-1/wal/wals/92a7317794fa49768b3146e33630bfc9
I20260619 16:41:40.404098 7862 ts_tablet_manager.cc:1939] T 92a7317794fa49768b3146e33630bfc9 P da90af8c16574881832a55a5861d2c8f: Deleting consensus metadata
I20260619 16:41:40.405260 7279 catalog_manager.cc:4841] TS da90af8c16574881832a55a5861d2c8f (127.0.98.2:41495): tablet 92a7317794fa49768b3146e33630bfc9 (table test-table [id=f27ccc08f54b421c9ada18670235d196]) successfully deleted
I20260619 16:41:40.408075 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 7480
I20260619 16:41:40.409602 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:40.409172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41841 (local address 127.0.0.1:45340)
0619 16:41:40.409375 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:40.409477 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 16:41:40.409905 7279 catalog_manager.cc:4584] TS 715aa0a6a1334d59a2680f0b8cef1c85 (127.0.98.1:41841): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 92a7317794fa49768b3146e33630bfc9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41841: connect: Connection refused (error 111)
I20260619 16:41:40.416435 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 7616
I20260619 16:41:40.425132 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 6965
I20260619 16:41:40.431823 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 7074
2026-06-19T16:41:40Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5948 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:41:40 dist-test-slave-03cd krb5kdc[7866](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:41:40 dist-test-slave-03cd krb5kdc[7866](info): set up 2 sockets
Jun 19 16:41:40 dist-test-slave-03cd krb5kdc[7866](info): commencing operation
krb5kdc: starting...
W20260619 16:41:41.521080 392 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:41:41 dist-test-slave-03cd krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:41:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:41:41Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260619 16:41:41.667886 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39229
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:41.802871 7879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:41.803256 7879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:41.803304 7879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:41.808029 7879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:41:41.808135 7879 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 16:41:41.808154 7879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:41.808174 7879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:41.808193 7879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:41:41.808209 7879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:41:41.814723 7879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39229
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:41.816284 7879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:41.817642 7879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:41.828583 7885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:41.828574 7884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:41.828574 7887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:41.829321 7879 server_base.cc:1034] running on GCE node
I20260619 16:41:41.830400 7879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:41.832116 7879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:41.835444 7879 hybrid_clock.cc:648] HybridClock initialized: now 1781887301835392 us; error 86 us; skew 500 ppm
Jun 19 16:41:41 dist-test-slave-03cd krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887301, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:41.838702 7879 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260619 16:41:41.839816 7879 webserver.cc:460] Webserver started at http://127.0.98.62:34161/ using document root <none> and password file <none>
I20260619 16:41:41.840430 7879 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:41.840493 7879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:41.840683 7879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:41.842885 7879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "81e280c69e54404f838b915627e155a8"
format_stamp: "Formatted at 2026-06-19 16:41:41 on dist-test-slave-03cd"
server_key: "0847071693f6fc270458178baee41911"
server_key_iv: "bdf073a4a598983de51b920763b77341"
server_key_version: "encryptionkey@0"
I20260619 16:41:41.843446 7879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "81e280c69e54404f838b915627e155a8"
format_stamp: "Formatted at 2026-06-19 16:41:41 on dist-test-slave-03cd"
server_key: "0847071693f6fc270458178baee41911"
server_key_iv: "bdf073a4a598983de51b920763b77341"
server_key_version: "encryptionkey@0"
I20260619 16:41:41.848079 7879 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:41:41.851424 7894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:41.853384 7879 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:41:41.853534 7879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "81e280c69e54404f838b915627e155a8"
format_stamp: "Formatted at 2026-06-19 16:41:41 on dist-test-slave-03cd"
server_key: "0847071693f6fc270458178baee41911"
server_key_iv: "bdf073a4a598983de51b920763b77341"
server_key_version: "encryptionkey@0"
I20260619 16:41:41.853754 7879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:41.868193 7879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:41.872088 7879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:41.872323 7879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:41.882310 7879 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39229
I20260619 16:41:41.882383 7956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39229 every 8 connection(s)
I20260619 16:41:41.883400 7879 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:41:41.887264 7957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:41.887431 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 7879
I20260619 16:41:41.887558 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:41:41.887965 392 external_mini_cluster.cc:1428] Setting key 226d2d3cb9dcd60d2e723da184ce333b
I20260619 16:41:41.897028 7957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e280c69e54404f838b915627e155a8: Bootstrap starting.
Jun 19 16:41:41 dist-test-slave-03cd krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:41:41.901156 7957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81e280c69e54404f838b915627e155a8: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:41.902386 7957 log.cc:824] T 00000000000000000000000000000000 P 81e280c69e54404f838b915627e155a8: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:41.905623 7957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e280c69e54404f838b915627e155a8: No bootstrap required, opened a new log
I20260619 16:41:41.908250 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:41.889794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54452 (local address 127.0.98.62:39229)
0619 16:41:41.890301 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:41.890317 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:41.890359 (+ 42us) server_negotiation.cc:407] Connection header received
0619 16:41:41.891232 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:41.891251 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:41.891626 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:41.892048 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:41.893316 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:41.894473 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:41.896408 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:41.897057 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:41.897295 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:41.901296 (+ 4001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:41.901330 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:41.901360 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:41.901413 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:41.904521 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:41.905198 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:41.905213 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:41.905219 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:41.905356 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:41.905680 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:41.905683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:41.905687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:41.906330 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:41.906560 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:41.906901 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 16:41:41.907279 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":179,"threads_started":1}
I20260619 16:41:41.910234 7957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81e280c69e54404f838b915627e155a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e280c69e54404f838b915627e155a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39229 } }
I20260619 16:41:41.910665 7957 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81e280c69e54404f838b915627e155a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:41.910722 7957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81e280c69e54404f838b915627e155a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81e280c69e54404f838b915627e155a8, State: Initialized, Role: FOLLOWER
I20260619 16:41:41.911331 7957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81e280c69e54404f838b915627e155a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e280c69e54404f838b915627e155a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39229 } }
I20260619 16:41:41.911530 7957 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81e280c69e54404f838b915627e155a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:41:41.911618 7957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81e280c69e54404f838b915627e155a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:41:41.911814 7957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81e280c69e54404f838b915627e155a8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:41.913147 7957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81e280c69e54404f838b915627e155a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e280c69e54404f838b915627e155a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39229 } }
I20260619 16:41:41.913646 7957 leader_election.cc:302] T 00000000000000000000000000000000 P 81e280c69e54404f838b915627e155a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81e280c69e54404f838b915627e155a8; no voters:
I20260619 16:41:41.914098 7957 leader_election.cc:288] T 00000000000000000000000000000000 P 81e280c69e54404f838b915627e155a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:41:41.914361 7962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81e280c69e54404f838b915627e155a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:41.915077 7962 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81e280c69e54404f838b915627e155a8 [term 1 LEADER]: Becoming Leader. State: Replica: 81e280c69e54404f838b915627e155a8, State: Running, Role: LEADER
I20260619 16:41:41.915591 7962 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81e280c69e54404f838b915627e155a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e280c69e54404f838b915627e155a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39229 } }
I20260619 16:41:41.915956 7957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81e280c69e54404f838b915627e155a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:41:41.918071 7963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e280c69e54404f838b915627e155a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81e280c69e54404f838b915627e155a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e280c69e54404f838b915627e155a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39229 } } }
I20260619 16:41:41.918277 7963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e280c69e54404f838b915627e155a8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:41.918650 7964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e280c69e54404f838b915627e155a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81e280c69e54404f838b915627e155a8. Latest consensus state: current_term: 1 leader_uuid: "81e280c69e54404f838b915627e155a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e280c69e54404f838b915627e155a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39229 } } }
I20260619 16:41:41.919008 7976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:41:41.919152 7964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e280c69e54404f838b915627e155a8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:41.923132 7976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:41:41.932401 7976 catalog_manager.cc:1347] Generated new cluster ID: 118fe5fd838b4d4d832ea41760ffda2e
I20260619 16:41:41.932541 7976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:41:41.954515 7976 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:41:41.956430 7976 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:41:41.972926 7976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81e280c69e54404f838b915627e155a8: Generated new TSK 0
I20260619 16:41:41.974565 7976 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260619 16:41:42.019986 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39229
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:42.153594 7983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:42.154109 7983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:42.154258 7983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:42.159128 7983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:42.159238 7983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:42.159325 7983 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:41:42.159410 7983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:41:42.166108 7983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:42.167500 7983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:42.168803 7983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:42.179998 7991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:42.180017 7988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:42.180002 7989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:42.180388 7983 server_base.cc:1034] running on GCE node
I20260619 16:41:42.181304 7983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:42.182133 7983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:42.183413 7983 hybrid_clock.cc:648] HybridClock initialized: now 1781887302183374 us; error 56 us; skew 500 ppm
Jun 19 16:41:42 dist-test-slave-03cd krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887302, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:42.186784 7983 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 16:41:42.187934 7983 webserver.cc:460] Webserver started at http://127.0.98.1:45285/ using document root <none> and password file <none>
I20260619 16:41:42.188583 7983 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:42.188704 7983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:42.188977 7983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:42.191210 7983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-0/data/instance:
uuid: "a26293ac1ff74ce5a54b683d5f47954d"
format_stamp: "Formatted at 2026-06-19 16:41:42 on dist-test-slave-03cd"
server_key: "a8d15b9c8f7e6fa207f339df8b052164"
server_key_iv: "d8b6c29a840171d546e75f907acadc20"
server_key_version: "encryptionkey@0"
I20260619 16:41:42.191936 7983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a26293ac1ff74ce5a54b683d5f47954d"
format_stamp: "Formatted at 2026-06-19 16:41:42 on dist-test-slave-03cd"
server_key: "a8d15b9c8f7e6fa207f339df8b052164"
server_key_iv: "d8b6c29a840171d546e75f907acadc20"
server_key_version: "encryptionkey@0"
I20260619 16:41:42.196090 7983 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:41:42.198894 7998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:42.200435 7983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:41:42.200599 7983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "a26293ac1ff74ce5a54b683d5f47954d"
format_stamp: "Formatted at 2026-06-19 16:41:42 on dist-test-slave-03cd"
server_key: "a8d15b9c8f7e6fa207f339df8b052164"
server_key_iv: "d8b6c29a840171d546e75f907acadc20"
server_key_version: "encryptionkey@0"
I20260619 16:41:42.200740 7983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:42.224924 7983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:42.228509 7983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:42.228734 7983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:42.230290 7983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:42.230368 7983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:42.230448 7983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:42.230494 7983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:41:42 dist-test-slave-03cd krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887302, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:41:42.241968 7983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41255
I20260619 16:41:42.241986 8113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41255 every 8 connection(s)
I20260619 16:41:42.243207 7983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:41:42.246205 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:42.231426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54183 (local address 127.0.98.62:39229)
0619 16:41:42.231569 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:42.231573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:42.232761 (+ 1188us) server_negotiation.cc:407] Connection header received
0619 16:41:42.233988 (+ 1227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:42.233992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:42.234055 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:42.234172 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:42.235965 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.236881 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.238712 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.239084 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.239231 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:42.242298 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:42.242302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:42.242305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:42.242336 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:42.244195 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:42.244747 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:42.244750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:42.244751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:42.244859 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:42.245187 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:42.245189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:42.245190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:42.245378 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:42.245452 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:42.245956 (+ 504us) server_negotiation.cc:299] Negotiation successful
0619 16:41:42.246078 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:41:42.247123 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:42.232027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39229 (local address 127.0.98.1:54183)
0619 16:41:42.232588 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:42.232633 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:42.233731 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:42.234418 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:42.234430 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:42.234960 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:42.235768 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:42.235788 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.237054 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.237058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:42.238536 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:42.238573 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.239300 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.239307 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:42.239473 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:42.240222 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:42.240306 (+ 84us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:42.242067 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:42.244315 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:42.244320 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:42.244334 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:42.244631 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:42.244952 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:42.244955 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:42.244956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:42.245097 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:42.245474 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:42.245479 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:42.245727 (+ 248us) client_negotiation.cc:769] Sending connection context
0619 16:41:42.245897 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 16:41:42.246182 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":146,"threads_started":1}
I20260619 16:41:42.248618 8114 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39229
I20260619 16:41:42.248804 8114 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:42.249440 8114 heartbeater.cc:511] Master 127.0.98.62:39229 requested a full tablet report, sending...
I20260619 16:41:42.249552 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 7983
I20260619 16:41:42.249706 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:41:42.250012 392 external_mini_cluster.cc:1428] Setting key 82fb71b6a55445882dd913f5a12f0b4e
I20260619 16:41:42.251315 7910 ts_manager.cc:194] Registered new tserver with Master: a26293ac1ff74ce5a54b683d5f47954d (127.0.98.1:41255)
I20260619 16:41:42.252998 7910 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54183
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:41:42.262205 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:42.253833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59865 (local address 127.0.98.62:39229)
0619 16:41:42.253978 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:42.253983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:42.254240 (+ 257us) server_negotiation.cc:407] Connection header received
0619 16:41:42.254401 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:42.254405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:42.254455 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:42.254543 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:42.255028 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.255933 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.256894 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.257201 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.257318 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:42.258752 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:42.258756 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:42.258758 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:42.258785 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:42.260376 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:42.260833 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:42.260837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:42.260839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:42.260971 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:42.261268 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:42.261271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:42.261273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:42.261467 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:42.261572 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:42.261857 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 16:41:42.261990 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 16:41:42.262210 8120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:42.251087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39229 (local address 127.0.98.1:59865)
0619 16:41:42.254155 (+ 3068us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:42.254172 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:42.254291 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:42.254563 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:42.254566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:42.254795 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:42.254908 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:42.254914 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.256094 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.256097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:42.256743 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:42.256752 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.257331 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.257335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:42.257443 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:42.258118 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:42.258135 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:42.258604 (+ 469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:42.260506 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:42.260510 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:42.260513 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:42.260726 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:42.261068 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:42.261072 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:42.261074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:42.261168 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:42.261575 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:42.261580 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:42.261680 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 16:41:42.261796 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 16:41:42.261994 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2981,"thread_start_us":152,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260619 16:41:42.283011 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39229
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:42.403225 8123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:42.403587 8123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:42.403720 8123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:42.408301 8123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:42.408398 8123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:42.408468 8123 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:41:42.408553 8123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:41:42.414296 8123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:42.415527 8123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:42.416747 8123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:42.427019 8129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:42.427003 8128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:42.427037 8131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:42.427484 8123 server_base.cc:1034] running on GCE node
I20260619 16:41:42.428006 8123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:42.428712 8123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:42.429906 8123 hybrid_clock.cc:648] HybridClock initialized: now 1781887302429894 us; error 44 us; skew 500 ppm
Jun 19 16:41:42 dist-test-slave-03cd krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887302, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:42.432993 8123 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 16:41:42.433990 8123 webserver.cc:460] Webserver started at http://127.0.98.2:33851/ using document root <none> and password file <none>
I20260619 16:41:42.434578 8123 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:42.434671 8123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:42.434917 8123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:42.437136 8123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-1/data/instance:
uuid: "b170ecc3958549ba8bf604b1b301c9f5"
format_stamp: "Formatted at 2026-06-19 16:41:42 on dist-test-slave-03cd"
server_key: "449bb5b3fffc281bfa464c965f7c0d37"
server_key_iv: "c8e6cef793b0843288c277b0e979e830"
server_key_version: "encryptionkey@0"
I20260619 16:41:42.437764 8123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b170ecc3958549ba8bf604b1b301c9f5"
format_stamp: "Formatted at 2026-06-19 16:41:42 on dist-test-slave-03cd"
server_key: "449bb5b3fffc281bfa464c965f7c0d37"
server_key_iv: "c8e6cef793b0843288c277b0e979e830"
server_key_version: "encryptionkey@0"
I20260619 16:41:42.441955 8123 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:41:42.444767 8138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:42.446303 8123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:41:42.446508 8123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "b170ecc3958549ba8bf604b1b301c9f5"
format_stamp: "Formatted at 2026-06-19 16:41:42 on dist-test-slave-03cd"
server_key: "449bb5b3fffc281bfa464c965f7c0d37"
server_key_iv: "c8e6cef793b0843288c277b0e979e830"
server_key_version: "encryptionkey@0"
I20260619 16:41:42.446650 8123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:42.474344 8123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:42.477996 8123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:42.478236 8123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:42.479804 8123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:42.479883 8123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:42.479945 8123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:42.479974 8123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:41:42 dist-test-slave-03cd krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887302, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:41:42.491117 8123 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43469
I20260619 16:41:42.491187 8253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43469 every 8 connection(s)
I20260619 16:41:42.492161 8123 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:41:42.495625 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:42.480855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44885 (local address 127.0.98.62:39229)
0619 16:41:42.480990 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:42.480994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:42.482058 (+ 1064us) server_negotiation.cc:407] Connection header received
0619 16:41:42.483336 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:42.483340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:42.483404 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:42.483498 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:42.485297 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.486225 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.488206 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.488435 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.488529 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:42.491812 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:42.491816 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:42.491818 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:42.491850 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:42.493521 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:42.494021 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:42.494024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:42.494025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:42.494118 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:42.494528 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:42.494531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:42.494533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:42.494732 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:42.494824 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:42.495373 (+ 549us) server_negotiation.cc:299] Negotiation successful
0619 16:41:42.495492 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:41:42.496447 8147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:42.481353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39229 (local address 127.0.98.2:44885)
0619 16:41:42.481886 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:42.481929 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:42.483079 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:42.483731 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:42.483743 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:42.484285 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:42.485068 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:42.485095 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.486431 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.486435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:42.488071 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:42.488082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.488611 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.488616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:42.488845 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:42.489660 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:42.489685 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:42.491585 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:42.493632 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:42.493637 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:42.493647 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:42.493914 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:42.494202 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:42.494205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:42.494206 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:42.494387 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:42.494827 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:42.494832 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:42.495084 (+ 252us) client_negotiation.cc:769] Sending connection context
0619 16:41:42.495234 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 16:41:42.495533 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":136,"threads_started":1}
I20260619 16:41:42.497997 8254 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39229
I20260619 16:41:42.498208 8254 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:42.498834 8254 heartbeater.cc:511] Master 127.0.98.62:39229 requested a full tablet report, sending...
I20260619 16:41:42.499987 7911 ts_manager.cc:194] Registered new tserver with Master: b170ecc3958549ba8bf604b1b301c9f5 (127.0.98.2:43469)
I20260619 16:41:42.500465 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 8123
I20260619 16:41:42.500584 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:41:42.500880 392 external_mini_cluster.cc:1428] Setting key 6eb19f99d5d60231d06c66bc7556271d
I20260619 16:41:42.500880 7911 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44885
I20260619 16:41:42.508782 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:42.500542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48417 (local address 127.0.98.62:39229)
0619 16:41:42.500670 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:42.500675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:42.500959 (+ 284us) server_negotiation.cc:407] Connection header received
0619 16:41:42.501145 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:42.501148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:42.501197 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:42.501284 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:42.501976 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.502679 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.503536 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.503965 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.504086 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:42.505423 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:42.505426 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:42.505430 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:42.505460 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:42.507051 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:42.507481 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:42.507486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:42.507488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:42.507581 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:42.507888 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:42.507893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:42.507895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:42.508120 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:42.508226 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:42.508493 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 16:41:42.508615 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:41:42.508816 8260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:42.500440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39229 (local address 127.0.98.2:48417)
0619 16:41:42.500869 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:42.500885 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:42.501033 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:42.501393 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:42.501396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:42.501622 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:42.501843 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:42.501852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.502827 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.502830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:42.503412 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:42.503421 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.504093 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.504097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:42.504187 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:42.504891 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:42.504907 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:42.505269 (+ 362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:42.507164 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:42.507168 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:42.507170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:42.507383 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:42.507694 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:42.507700 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:42.507702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:42.507790 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:42.508226 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:42.508229 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:42.508326 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 16:41:42.508438 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 16:41:42.508603 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":1,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 16:41:42.531288 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39229
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:42.654183 8263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:42.654564 8263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:42.654666 8263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:42.659611 8263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:42.659766 8263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:42.659847 8263 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:41:42.659936 8263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:41:42.667246 8263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:42.668516 8263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:42.669631 8263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:42.680657 8271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:42.680851 8269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:42.680910 8268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:42.681864 8263 server_base.cc:1034] running on GCE node
I20260619 16:41:42.682528 8263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:42.683233 8263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:42.686658 8263 hybrid_clock.cc:648] HybridClock initialized: now 1781887302686593 us; error 47 us; skew 500 ppm
Jun 19 16:41:42 dist-test-slave-03cd krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887302, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:42.691074 8263 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 16:41:42.692253 8263 webserver.cc:460] Webserver started at http://127.0.98.3:37909/ using document root <none> and password file <none>
I20260619 16:41:42.692842 8263 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:42.692909 8263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:42.693091 8263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:42.695299 8263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-2/data/instance:
uuid: "61292455bfa049e38d0cbda9e3d49542"
format_stamp: "Formatted at 2026-06-19 16:41:42 on dist-test-slave-03cd"
server_key: "4ecbd8a6f5d148ff086abbd30feea7d7"
server_key_iv: "903188e10a53823631c0c5fcf0e65048"
server_key_version: "encryptionkey@0"
I20260619 16:41:42.696336 8263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-2/wal/instance:
uuid: "61292455bfa049e38d0cbda9e3d49542"
format_stamp: "Formatted at 2026-06-19 16:41:42 on dist-test-slave-03cd"
server_key: "4ecbd8a6f5d148ff086abbd30feea7d7"
server_key_iv: "903188e10a53823631c0c5fcf0e65048"
server_key_version: "encryptionkey@0"
I20260619 16:41:42.701572 8263 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:41:42.704947 8278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:42.707156 8263 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:41:42.707363 8263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "61292455bfa049e38d0cbda9e3d49542"
format_stamp: "Formatted at 2026-06-19 16:41:42 on dist-test-slave-03cd"
server_key: "4ecbd8a6f5d148ff086abbd30feea7d7"
server_key_iv: "903188e10a53823631c0c5fcf0e65048"
server_key_version: "encryptionkey@0"
I20260619 16:41:42.707531 8263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:42.719293 8263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:42.723289 8263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:42.723533 8263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:42.725049 8263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:42.725100 8263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:42.725142 8263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:42.725204 8263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:41:42 dist-test-slave-03cd krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887302, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:41:42.736218 8263 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44575
I20260619 16:41:42.736280 8393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44575 every 8 connection(s)
I20260619 16:41:42.737396 8263 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:41:42.739266 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 8263
I20260619 16:41:42.739382 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:41:42.739835 392 external_mini_cluster.cc:1428] Setting key 64e1f28cdffb62d5224091f925c48dfd
I20260619 16:41:42.741678 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:42.726314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44959 (local address 127.0.98.62:39229)
0619 16:41:42.726464 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:42.726468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:42.727618 (+ 1150us) server_negotiation.cc:407] Connection header received
0619 16:41:42.728837 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:42.728841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:42.728909 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:42.728999 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:42.730680 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.731590 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.732939 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.733229 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.733363 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:42.736142 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:42.736146 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:42.736149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:42.736183 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:42.738210 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:42.739217 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:42.739221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:42.739222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:42.739350 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:42.739891 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:42.739895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:42.739897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:42.740169 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:42.740525 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:42.741126 (+ 601us) server_negotiation.cc:299] Negotiation successful
0619 16:41:42.741422 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:41:42.742513 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:42.726815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39229 (local address 127.0.98.3:44959)
0619 16:41:42.727415 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:42.727483 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:42.728591 (+ 1108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:42.729197 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:42.729205 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:42.729723 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:42.730506 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:42.730530 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.731796 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.731800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:42.732791 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:42.732802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.733417 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.733422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:42.733621 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:42.734312 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:42.734329 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:42.735951 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:42.738466 (+ 2515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:42.738473 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:42.738492 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:42.739070 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:42.739462 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:42.739468 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:42.739474 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:42.739741 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:42.740300 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:42.740309 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:42.740683 (+ 374us) client_negotiation.cc:769] Sending connection context
0619 16:41:42.740909 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 16:41:42.741361 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":142,"threads_started":1}
I20260619 16:41:42.743958 8394 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39229
I20260619 16:41:42.744195 8394 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:42.745131 8394 heartbeater.cc:511] Master 127.0.98.62:39229 requested a full tablet report, sending...
I20260619 16:41:42.746395 7911 ts_manager.cc:194] Registered new tserver with Master: 61292455bfa049e38d0cbda9e3d49542 (127.0.98.3:44575)
I20260619 16:41:42.747325 7911 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44959
I20260619 16:41:42.756103 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:41:42.756346 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:42.746349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39229 (local address 127.0.98.3:59095)
0619 16:41:42.746708 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:42.746737 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:42.746862 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:42.747121 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:42.747124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:42.747410 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:42.747557 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:42.747567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.748532 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.748535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:42.749262 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:42.749272 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.749761 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.749764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:42.749854 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:42.750477 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:42.750492 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:42.750888 (+ 396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:42.754145 (+ 3257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:42.754151 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:42.754153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:42.754415 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:42.754980 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:42.754984 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:42.754986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:42.755108 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:42.755657 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:42.755693 (+ 36us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:42.755833 (+ 140us) client_negotiation.cc:769] Sending connection context
0619 16:41:42.755963 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 16:41:42.756132 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":124,"threads_started":1}
I20260619 16:41:42.756464 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:42.746470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59095 (local address 127.0.98.62:39229)
0619 16:41:42.746598 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:42.746605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:42.746866 (+ 261us) server_negotiation.cc:407] Connection header received
0619 16:41:42.746960 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:42.746963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:42.747004 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:42.747076 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:42.747739 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.748380 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.749385 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.749635 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.749751 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:42.751040 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:42.751045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:42.751053 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:42.751078 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:42.753997 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:42.754590 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:42.754594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:42.754596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:42.754768 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:42.755225 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:42.755229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:42.755231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:42.755486 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:42.755626 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:42.756103 (+ 477us) server_negotiation.cc:299] Negotiation successful
0619 16:41:42.756325 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 16:41:42 dist-test-slave-03cd krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887302, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:41:42 dist-test-slave-03cd krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:42 dist-test-slave-03cd krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 16:41:42.785333 7960 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:42.778282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54460 (local address 127.0.98.62:39229)
0619 16:41:42.778447 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:42.778451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:42.778466 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:41:42.778513 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:42.778516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:42.778558 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:42.778676 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:42.779402 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.780147 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.781081 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.781327 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.781420 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:42.785149 (+ 3729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54460: BlockingRecv error: recv got EOF from 127.0.0.1:54460 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 16:41:42 dist-test-slave-03cd krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887302, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:41:42.796178 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:42.786839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54464 (local address 127.0.98.62:39229)
0619 16:41:42.786959 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:42.786962 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:42.787095 (+ 133us) server_negotiation.cc:407] Connection header received
0619 16:41:42.787199 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:42.787201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:42.787242 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:42.787313 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:42.788049 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.788611 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.789416 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.789659 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.789742 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:42.792543 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:42.792546 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:42.792548 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:42.792575 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:42.794052 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:42.794740 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:42.794744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:42.794745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:42.794827 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:42.795189 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:42.795191 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:42.795192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:42.795404 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:42.795539 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:42.795883 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 16:41:42.795997 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 16:41:42.799465 7911 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:41:42.802845 7911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:41:42.818598 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:42.813140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50444 (local address 127.0.98.1:41255)
0619 16:41:42.813951 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:42.813958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:42.814188 (+ 230us) server_negotiation.cc:407] Connection header received
0619 16:41:42.814253 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:42.814256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:42.814392 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:42.814544 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:42.814812 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.815600 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.817191 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.817867 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.817997 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:42.818159 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:42.818323 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 16:41:42.818429 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":114,"threads_started":1}
I20260619 16:41:42.818616 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:42.812897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41255 (local address 127.0.0.1:50444)
0619 16:41:42.814000 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:42.814014 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:42.814093 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:42.814538 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:42.814544 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:42.814565 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:42.814698 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:42.814705 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.815769 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.815772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:42.817048 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:42.817060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.817999 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.818002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:42.818106 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:42.818141 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 16:41:42.818282 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 16:41:42.818420 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1035,"mutex_wait_us":492,"thread_start_us":98,"threads_started":1}
I20260619 16:41:42.819769 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:42.812570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44575 (local address 127.0.0.1:44182)
0619 16:41:42.813231 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:42.813289 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:42.813422 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:42.814687 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:42.814691 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:42.814712 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:42.814879 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:42.814886 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.816571 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.816574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:42.817694 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:42.817703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.819161 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.819166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:42.819254 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:42.819277 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:41:42.819340 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 16:41:42.819405 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":42,"spinlock_wait_cycles":2304,"thread_start_us":99,"threads_started":1}
I20260619 16:41:42.819603 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:42.812666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44182 (local address 127.0.98.3:44575)
0619 16:41:42.814244 (+ 1578us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:42.814256 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:42.814278 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:41:42.814351 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:42.814356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:42.814527 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:42.814870 (+ 343us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:42.815021 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.816447 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.817824 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.819036 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.819154 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:42.819312 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:42.819363 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 16:41:42.819454 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1449,"thread_start_us":134,"threads_started":1}
I20260619 16:41:42.820649 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:42.813328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43469 (local address 127.0.0.1:37470)
0619 16:41:42.813724 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:42.813737 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:42.813827 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:42.814706 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:42.814709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:42.814722 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:42.814891 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:42.814900 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.816575 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.816577 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:42.818867 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:42.818877 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.820214 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.820219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:42.820318 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:42.820338 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 16:41:42.820399 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 16:41:42.820477 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":65,"thread_start_us":121,"threads_started":1}
I20260619 16:41:42.821039 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:42.813518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37470 (local address 127.0.98.2:43469)
0619 16:41:42.813909 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:42.813925 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:42.813952 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:41:42.814038 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:42.814046 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:42.814364 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:42.814529 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:42.815032 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.816418 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.819019 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:42.819853 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:42.819972 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:42.820141 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:42.820811 (+ 670us) server_negotiation.cc:299] Negotiation successful
0619 16:41:42.820912 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":109,"threads_started":1}
I20260619 16:41:42.822593 8048 tablet_service.cc:1467] Processing CreateTablet for tablet 4fdc20ef175941fbad8b2f3e5197ab55 (DEFAULT_TABLE table=test-table [id=7b75a04e94b94e0fb5c95b501747bfd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:42.823992 8048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fdc20ef175941fbad8b2f3e5197ab55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:42.824124 8328 tablet_service.cc:1467] Processing CreateTablet for tablet 4fdc20ef175941fbad8b2f3e5197ab55 (DEFAULT_TABLE table=test-table [id=7b75a04e94b94e0fb5c95b501747bfd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:42.824664 8188 tablet_service.cc:1467] Processing CreateTablet for tablet 4fdc20ef175941fbad8b2f3e5197ab55 (DEFAULT_TABLE table=test-table [id=7b75a04e94b94e0fb5c95b501747bfd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:42.825269 8328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fdc20ef175941fbad8b2f3e5197ab55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:42.825732 8188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fdc20ef175941fbad8b2f3e5197ab55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:42.830529 8422 tablet_bootstrap.cc:492] T 4fdc20ef175941fbad8b2f3e5197ab55 P a26293ac1ff74ce5a54b683d5f47954d: Bootstrap starting.
I20260619 16:41:42.833374 8422 tablet_bootstrap.cc:654] T 4fdc20ef175941fbad8b2f3e5197ab55 P a26293ac1ff74ce5a54b683d5f47954d: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:42.834458 8422 log.cc:824] T 4fdc20ef175941fbad8b2f3e5197ab55 P a26293ac1ff74ce5a54b683d5f47954d: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:42.835016 8424 tablet_bootstrap.cc:492] T 4fdc20ef175941fbad8b2f3e5197ab55 P 61292455bfa049e38d0cbda9e3d49542: Bootstrap starting.
I20260619 16:41:42.835142 8423 tablet_bootstrap.cc:492] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5: Bootstrap starting.
I20260619 16:41:42.837373 8424 tablet_bootstrap.cc:654] T 4fdc20ef175941fbad8b2f3e5197ab55 P 61292455bfa049e38d0cbda9e3d49542: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:42.837585 8423 tablet_bootstrap.cc:654] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:42.838184 8424 log.cc:824] T 4fdc20ef175941fbad8b2f3e5197ab55 P 61292455bfa049e38d0cbda9e3d49542: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:42.838347 8423 log.cc:824] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:42.840662 8424 tablet_bootstrap.cc:492] T 4fdc20ef175941fbad8b2f3e5197ab55 P 61292455bfa049e38d0cbda9e3d49542: No bootstrap required, opened a new log
I20260619 16:41:42.840755 8422 tablet_bootstrap.cc:492] T 4fdc20ef175941fbad8b2f3e5197ab55 P a26293ac1ff74ce5a54b683d5f47954d: No bootstrap required, opened a new log
I20260619 16:41:42.840946 8424 ts_tablet_manager.cc:1397] T 4fdc20ef175941fbad8b2f3e5197ab55 P 61292455bfa049e38d0cbda9e3d49542: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260619 16:41:42.840992 8422 ts_tablet_manager.cc:1397] T 4fdc20ef175941fbad8b2f3e5197ab55 P a26293ac1ff74ce5a54b683d5f47954d: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260619 16:41:42.844146 8424 raft_consensus.cc:348] T 4fdc20ef175941fbad8b2f3e5197ab55 P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } }
I20260619 16:41:42.844468 8424 raft_consensus.cc:374] T 4fdc20ef175941fbad8b2f3e5197ab55 P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:42.844857 8423 tablet_bootstrap.cc:492] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5: No bootstrap required, opened a new log
I20260619 16:41:42.844717 8424 raft_consensus.cc:729] T 4fdc20ef175941fbad8b2f3e5197ab55 P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61292455bfa049e38d0cbda9e3d49542, State: Initialized, Role: FOLLOWER
I20260619 16:41:42.845094 8423 ts_tablet_manager.cc:1397] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260619 16:41:42.845417 8424 consensus_queue.cc:260] T 4fdc20ef175941fbad8b2f3e5197ab55 P 61292455bfa049e38d0cbda9e3d49542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } }
I20260619 16:41:42.845402 8422 raft_consensus.cc:348] T 4fdc20ef175941fbad8b2f3e5197ab55 P a26293ac1ff74ce5a54b683d5f47954d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } }
I20260619 16:41:42.845649 8422 raft_consensus.cc:374] T 4fdc20ef175941fbad8b2f3e5197ab55 P a26293ac1ff74ce5a54b683d5f47954d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:42.845705 8422 raft_consensus.cc:729] T 4fdc20ef175941fbad8b2f3e5197ab55 P a26293ac1ff74ce5a54b683d5f47954d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a26293ac1ff74ce5a54b683d5f47954d, State: Initialized, Role: FOLLOWER
I20260619 16:41:42.846150 8422 consensus_queue.cc:260] T 4fdc20ef175941fbad8b2f3e5197ab55 P a26293ac1ff74ce5a54b683d5f47954d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } }
I20260619 16:41:42.846765 8424 ts_tablet_manager.cc:1428] T 4fdc20ef175941fbad8b2f3e5197ab55 P 61292455bfa049e38d0cbda9e3d49542: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260619 16:41:42.847152 8394 heartbeater.cc:503] Master 127.0.98.62:39229 was elected leader, sending a full tablet report...
I20260619 16:41:42.847934 8422 ts_tablet_manager.cc:1428] T 4fdc20ef175941fbad8b2f3e5197ab55 P a26293ac1ff74ce5a54b683d5f47954d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 16:41:42.848403 8114 heartbeater.cc:503] Master 127.0.98.62:39229 was elected leader, sending a full tablet report...
I20260619 16:41:42.849288 8423 raft_consensus.cc:348] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } }
I20260619 16:41:42.849608 8423 raft_consensus.cc:374] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:42.849674 8423 raft_consensus.cc:729] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b170ecc3958549ba8bf604b1b301c9f5, State: Initialized, Role: FOLLOWER
I20260619 16:41:42.850142 8423 consensus_queue.cc:260] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } }
I20260619 16:41:42.850922 8254 heartbeater.cc:503] Master 127.0.98.62:39229 was elected leader, sending a full tablet report...
I20260619 16:41:42.851380 8423 ts_tablet_manager.cc:1428] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260619 16:41:42.989194 8395 tablet.cc:2369] T 4fdc20ef175941fbad8b2f3e5197ab55 P 61292455bfa049e38d0cbda9e3d49542: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:41:42.993861 8255 tablet.cc:2369] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:41:42.994779 8115 tablet.cc:2369] T 4fdc20ef175941fbad8b2f3e5197ab55 P a26293ac1ff74ce5a54b683d5f47954d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:41:43.136116 8430 raft_consensus.cc:482] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:41:43.136439 8430 raft_consensus.cc:504] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } }
I20260619 16:41:43.137723 8430 leader_election.cc:288] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61292455bfa049e38d0cbda9e3d49542 (127.0.98.3:44575), a26293ac1ff74ce5a54b683d5f47954d (127.0.98.1:41255)
I20260619 16:41:43.142800 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:43.138384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57329 (local address 127.0.98.1:41255)
0619 16:41:43.138677 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:43.138682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:43.138948 (+ 266us) server_negotiation.cc:407] Connection header received
0619 16:41:43.139109 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:43.139112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:43.139170 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:43.139280 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:43.139629 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.140208 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.141677 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.142092 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.142187 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:43.142292 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:43.142546 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 16:41:43.142649 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185}
I20260619 16:41:43.142855 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:43.138309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33123 (local address 127.0.98.3:44575)
0619 16:41:43.138618 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:43.138625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:43.138651 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:41:43.138881 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:43.138886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:43.138969 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:43.139067 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:43.139560 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.140252 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.141605 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.142107 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.142258 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:43.142342 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:43.142610 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 16:41:43.142704 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260619 16:41:43.142874 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:43.138165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41255 (local address 127.0.98.2:57329)
0619 16:41:43.138750 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:43.138766 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:43.138930 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:43.139310 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:43.139312 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:43.139328 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:43.139493 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:43.139501 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.140352 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.140360 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:43.141488 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:43.141500 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.142252 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.142256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:43.142373 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:43.142425 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 16:41:43.142500 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 16:41:43.142639 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":151,"thread_start_us":112,"threads_started":1}
I20260619 16:41:43.142882 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:43.138165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44575 (local address 127.0.98.2:33123)
0619 16:41:43.138530 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:43.138550 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:43.138718 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:43.139136 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:43.139139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:43.139163 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:43.139360 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:43.139370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.140451 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.140454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:43.141456 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:43.141465 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.142292 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.142297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:43.142435 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:43.142459 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 16:41:43.142571 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 16:41:43.142665 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":31,"thread_start_us":131,"threads_started":1}
I20260619 16:41:43.143990 8348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fdc20ef175941fbad8b2f3e5197ab55" candidate_uuid: "b170ecc3958549ba8bf604b1b301c9f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61292455bfa049e38d0cbda9e3d49542" is_pre_election: true
I20260619 16:41:43.144050 8068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fdc20ef175941fbad8b2f3e5197ab55" candidate_uuid: "b170ecc3958549ba8bf604b1b301c9f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a26293ac1ff74ce5a54b683d5f47954d" is_pre_election: true
I20260619 16:41:43.144439 8348 raft_consensus.cc:2454] T 4fdc20ef175941fbad8b2f3e5197ab55 P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b170ecc3958549ba8bf604b1b301c9f5 in term 0.
I20260619 16:41:43.144538 8068 raft_consensus.cc:2454] T 4fdc20ef175941fbad8b2f3e5197ab55 P a26293ac1ff74ce5a54b683d5f47954d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b170ecc3958549ba8bf604b1b301c9f5 in term 0.
I20260619 16:41:43.145025 8139 leader_election.cc:302] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61292455bfa049e38d0cbda9e3d49542, b170ecc3958549ba8bf604b1b301c9f5; no voters:
I20260619 16:41:43.145433 8430 raft_consensus.cc:2781] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:41:43.145551 8430 raft_consensus.cc:482] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:41:43.145638 8430 raft_consensus.cc:3037] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:43.147040 8430 raft_consensus.cc:504] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } }
I20260619 16:41:43.147573 8430 leader_election.cc:288] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5 [CANDIDATE]: Term 1 election: Requested vote from peers 61292455bfa049e38d0cbda9e3d49542 (127.0.98.3:44575), a26293ac1ff74ce5a54b683d5f47954d (127.0.98.1:41255)
I20260619 16:41:43.148200 8068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fdc20ef175941fbad8b2f3e5197ab55" candidate_uuid: "b170ecc3958549ba8bf604b1b301c9f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a26293ac1ff74ce5a54b683d5f47954d"
I20260619 16:41:43.148200 8348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fdc20ef175941fbad8b2f3e5197ab55" candidate_uuid: "b170ecc3958549ba8bf604b1b301c9f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61292455bfa049e38d0cbda9e3d49542"
I20260619 16:41:43.148406 8068 raft_consensus.cc:3037] T 4fdc20ef175941fbad8b2f3e5197ab55 P a26293ac1ff74ce5a54b683d5f47954d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:43.148388 8348 raft_consensus.cc:3037] T 4fdc20ef175941fbad8b2f3e5197ab55 P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:43.150315 8068 raft_consensus.cc:2454] T 4fdc20ef175941fbad8b2f3e5197ab55 P a26293ac1ff74ce5a54b683d5f47954d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b170ecc3958549ba8bf604b1b301c9f5 in term 1.
I20260619 16:41:43.150786 8348 raft_consensus.cc:2454] T 4fdc20ef175941fbad8b2f3e5197ab55 P 61292455bfa049e38d0cbda9e3d49542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b170ecc3958549ba8bf604b1b301c9f5 in term 1.
I20260619 16:41:43.150811 8141 leader_election.cc:302] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a26293ac1ff74ce5a54b683d5f47954d, b170ecc3958549ba8bf604b1b301c9f5; no voters:
I20260619 16:41:43.151093 8430 raft_consensus.cc:2781] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:43.151393 8430 raft_consensus.cc:686] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5 [term 1 LEADER]: Becoming Leader. State: Replica: b170ecc3958549ba8bf604b1b301c9f5, State: Running, Role: LEADER
I20260619 16:41:43.151950 8430 consensus_queue.cc:237] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } }
I20260619 16:41:43.156353 7910 catalog_manager.cc:5495] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b170ecc3958549ba8bf604b1b301c9f5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b170ecc3958549ba8bf604b1b301c9f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } health_report { overall_health: UNKNOWN } } }
I20260619 16:41:43.196201 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:43.191489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37474 (local address 127.0.98.2:43469)
0619 16:41:43.191626 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:43.191629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:43.191737 (+ 108us) server_negotiation.cc:407] Connection header received
0619 16:41:43.191919 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:43.191922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:43.191981 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:43.192089 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:41:43.192595 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.193145 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.194328 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.194563 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.194668 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:43.195027 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:41:43.195562 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:41:43.195806 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:43.195946 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 16:41:43.196038 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:41:43.204159 8068 raft_consensus.cc:1261] T 4fdc20ef175941fbad8b2f3e5197ab55 P a26293ac1ff74ce5a54b683d5f47954d [term 1 FOLLOWER]: Refusing update from remote peer b170ecc3958549ba8bf604b1b301c9f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:43.204313 8348 raft_consensus.cc:1261] T 4fdc20ef175941fbad8b2f3e5197ab55 P 61292455bfa049e38d0cbda9e3d49542 [term 1 FOLLOWER]: Refusing update from remote peer b170ecc3958549ba8bf604b1b301c9f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:43.205392 8430 consensus_queue.cc:1035] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:43.205711 8433 consensus_queue.cc:1035] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:43.217247 8438 mvcc.cc:205] Tried to move back new op lower bound from 7298610393913163776 to 7298610393713676288. Current Snapshot: MvccSnapshot[applied={T|T < 7298610393913163776}]
I20260619 16:41:43.228379 7910 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:41:43.228583 7910 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:41:43.231189 7910 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81e280c69e54404f838b915627e155a8: Sending DeleteTablet for 3 replicas of tablet 4fdc20ef175941fbad8b2f3e5197ab55
I20260619 16:41:43.232044 8328 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fdc20ef175941fbad8b2f3e5197ab55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44182
I20260619 16:41:43.232364 8048 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fdc20ef175941fbad8b2f3e5197ab55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50444
I20260619 16:41:43.232560 8446 tablet_replica.cc:330] T 4fdc20ef175941fbad8b2f3e5197ab55 P 61292455bfa049e38d0cbda9e3d49542: stopping tablet replica
I20260619 16:41:43.232926 8446 raft_consensus.cc:2229] T 4fdc20ef175941fbad8b2f3e5197ab55 P 61292455bfa049e38d0cbda9e3d49542 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:41:43.232952 8188 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fdc20ef175941fbad8b2f3e5197ab55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37470
I20260619 16:41:43.233283 8446 raft_consensus.cc:2258] T 4fdc20ef175941fbad8b2f3e5197ab55 P 61292455bfa049e38d0cbda9e3d49542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:43.233407 8448 tablet_replica.cc:330] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5: stopping tablet replica
I20260619 16:41:43.233664 8448 raft_consensus.cc:2229] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:41:43.234068 8448 raft_consensus.cc:2258] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:43.234630 7910 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 16:41:43.234948 8446 ts_tablet_manager.cc:1905] T 4fdc20ef175941fbad8b2f3e5197ab55 P 61292455bfa049e38d0cbda9e3d49542: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:43.235260 8448 ts_tablet_manager.cc:1905] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 16:41:43.235278 7910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:41:43.237530 8447 tablet_replica.cc:330] T 4fdc20ef175941fbad8b2f3e5197ab55 P a26293ac1ff74ce5a54b683d5f47954d: stopping tablet replica
I20260619 16:41:43.238016 8448 ts_tablet_manager.cc:1918] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:41:43.238131 8448 log.cc:1196] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-1/wal/wals/4fdc20ef175941fbad8b2f3e5197ab55
I20260619 16:41:43.238283 8447 raft_consensus.cc:2229] T 4fdc20ef175941fbad8b2f3e5197ab55 P a26293ac1ff74ce5a54b683d5f47954d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:41:43.238480 8448 ts_tablet_manager.cc:1939] T 4fdc20ef175941fbad8b2f3e5197ab55 P b170ecc3958549ba8bf604b1b301c9f5: Deleting consensus metadata
I20260619 16:41:43.238837 8447 raft_consensus.cc:2258] T 4fdc20ef175941fbad8b2f3e5197ab55 P a26293ac1ff74ce5a54b683d5f47954d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:43.239789 7895 catalog_manager.cc:4841] TS b170ecc3958549ba8bf604b1b301c9f5 (127.0.98.2:43469): tablet 4fdc20ef175941fbad8b2f3e5197ab55 (table test-table [id=7b75a04e94b94e0fb5c95b501747bfd7]) successfully deleted
I20260619 16:41:43.240684 8446 ts_tablet_manager.cc:1918] T 4fdc20ef175941fbad8b2f3e5197ab55 P 61292455bfa049e38d0cbda9e3d49542: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:41:43.241722 8446 log.cc:1196] T 4fdc20ef175941fbad8b2f3e5197ab55 P 61292455bfa049e38d0cbda9e3d49542: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-2/wal/wals/4fdc20ef175941fbad8b2f3e5197ab55
I20260619 16:41:43.242296 8446 ts_tablet_manager.cc:1939] T 4fdc20ef175941fbad8b2f3e5197ab55 P 61292455bfa049e38d0cbda9e3d49542: Deleting consensus metadata
I20260619 16:41:43.242409 8447 ts_tablet_manager.cc:1905] T 4fdc20ef175941fbad8b2f3e5197ab55 P a26293ac1ff74ce5a54b683d5f47954d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:43.245415 8447 ts_tablet_manager.cc:1918] T 4fdc20ef175941fbad8b2f3e5197ab55 P a26293ac1ff74ce5a54b683d5f47954d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:41:43.245517 8447 log.cc:1196] T 4fdc20ef175941fbad8b2f3e5197ab55 P a26293ac1ff74ce5a54b683d5f47954d: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-0/wal/wals/4fdc20ef175941fbad8b2f3e5197ab55
I20260619 16:41:43.245808 8447 ts_tablet_manager.cc:1939] T 4fdc20ef175941fbad8b2f3e5197ab55 P a26293ac1ff74ce5a54b683d5f47954d: Deleting consensus metadata
I20260619 16:41:43.245805 8048 tablet_service.cc:1467] Processing CreateTablet for tablet 1044700b8be343978ca6bcfe592a01c8 (DEFAULT_TABLE table=test-table [id=134dd24f47bc41548e41d8fdfd83ce3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:43.246275 8048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1044700b8be343978ca6bcfe592a01c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:43.247247 7897 catalog_manager.cc:4841] TS a26293ac1ff74ce5a54b683d5f47954d (127.0.98.1:41255): tablet 4fdc20ef175941fbad8b2f3e5197ab55 (table test-table [id=7b75a04e94b94e0fb5c95b501747bfd7]) successfully deleted
I20260619 16:41:43.249161 8422 tablet_bootstrap.cc:492] T 1044700b8be343978ca6bcfe592a01c8 P a26293ac1ff74ce5a54b683d5f47954d: Bootstrap starting.
I20260619 16:41:43.249868 7895 catalog_manager.cc:4841] TS 61292455bfa049e38d0cbda9e3d49542 (127.0.98.3:44575): tablet 4fdc20ef175941fbad8b2f3e5197ab55 (table test-table [id=7b75a04e94b94e0fb5c95b501747bfd7]) successfully deleted
I20260619 16:41:43.250248 8422 tablet_bootstrap.cc:654] T 1044700b8be343978ca6bcfe592a01c8 P a26293ac1ff74ce5a54b683d5f47954d: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:43.250820 8328 tablet_service.cc:1467] Processing CreateTablet for tablet 1044700b8be343978ca6bcfe592a01c8 (DEFAULT_TABLE table=test-table [id=134dd24f47bc41548e41d8fdfd83ce3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:43.251168 8328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1044700b8be343978ca6bcfe592a01c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:43.251765 8188 tablet_service.cc:1467] Processing CreateTablet for tablet 1044700b8be343978ca6bcfe592a01c8 (DEFAULT_TABLE table=test-table [id=134dd24f47bc41548e41d8fdfd83ce3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:43.252038 8188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1044700b8be343978ca6bcfe592a01c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:43.254380 8424 tablet_bootstrap.cc:492] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542: Bootstrap starting.
I20260619 16:41:43.254666 8423 tablet_bootstrap.cc:492] T 1044700b8be343978ca6bcfe592a01c8 P b170ecc3958549ba8bf604b1b301c9f5: Bootstrap starting.
I20260619 16:41:43.255625 8424 tablet_bootstrap.cc:654] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:43.256147 8423 tablet_bootstrap.cc:654] T 1044700b8be343978ca6bcfe592a01c8 P b170ecc3958549ba8bf604b1b301c9f5: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:43.258040 8424 tablet_bootstrap.cc:492] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542: No bootstrap required, opened a new log
I20260619 16:41:43.258179 8424 ts_tablet_manager.cc:1397] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 16:41:43.258364 8423 tablet_bootstrap.cc:492] T 1044700b8be343978ca6bcfe592a01c8 P b170ecc3958549ba8bf604b1b301c9f5: No bootstrap required, opened a new log
I20260619 16:41:43.258466 8423 ts_tablet_manager.cc:1397] T 1044700b8be343978ca6bcfe592a01c8 P b170ecc3958549ba8bf604b1b301c9f5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 16:41:43.258678 8424 raft_consensus.cc:348] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.258803 8424 raft_consensus.cc:374] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:43.258831 8424 raft_consensus.cc:729] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61292455bfa049e38d0cbda9e3d49542, State: Initialized, Role: FOLLOWER
I20260619 16:41:43.258961 8424 consensus_queue.cc:260] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.259060 8423 raft_consensus.cc:348] T 1044700b8be343978ca6bcfe592a01c8 P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.259207 8423 raft_consensus.cc:374] T 1044700b8be343978ca6bcfe592a01c8 P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:43.259243 8423 raft_consensus.cc:729] T 1044700b8be343978ca6bcfe592a01c8 P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b170ecc3958549ba8bf604b1b301c9f5, State: Initialized, Role: FOLLOWER
I20260619 16:41:43.259263 8424 ts_tablet_manager.cc:1428] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:41:43.259387 8423 consensus_queue.cc:260] T 1044700b8be343978ca6bcfe592a01c8 P b170ecc3958549ba8bf604b1b301c9f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.259780 8423 ts_tablet_manager.cc:1428] T 1044700b8be343978ca6bcfe592a01c8 P b170ecc3958549ba8bf604b1b301c9f5: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 16:41:43.260910 8422 tablet_bootstrap.cc:492] T 1044700b8be343978ca6bcfe592a01c8 P a26293ac1ff74ce5a54b683d5f47954d: No bootstrap required, opened a new log
I20260619 16:41:43.261037 8422 ts_tablet_manager.cc:1397] T 1044700b8be343978ca6bcfe592a01c8 P a26293ac1ff74ce5a54b683d5f47954d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260619 16:41:43.261570 8422 raft_consensus.cc:348] T 1044700b8be343978ca6bcfe592a01c8 P a26293ac1ff74ce5a54b683d5f47954d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.261716 8422 raft_consensus.cc:374] T 1044700b8be343978ca6bcfe592a01c8 P a26293ac1ff74ce5a54b683d5f47954d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:43.261749 8422 raft_consensus.cc:729] T 1044700b8be343978ca6bcfe592a01c8 P a26293ac1ff74ce5a54b683d5f47954d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a26293ac1ff74ce5a54b683d5f47954d, State: Initialized, Role: FOLLOWER
I20260619 16:41:43.261945 8422 consensus_queue.cc:260] T 1044700b8be343978ca6bcfe592a01c8 P a26293ac1ff74ce5a54b683d5f47954d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.262320 8422 ts_tablet_manager.cc:1428] T 1044700b8be343978ca6bcfe592a01c8 P a26293ac1ff74ce5a54b683d5f47954d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 16:41:43.330348 8428 raft_consensus.cc:482] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:41:43.330577 8428 raft_consensus.cc:504] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.331823 8428 leader_election.cc:288] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a26293ac1ff74ce5a54b683d5f47954d (127.0.98.1:41255), b170ecc3958549ba8bf604b1b301c9f5 (127.0.98.2:43469)
I20260619 16:41:43.337113 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:43.332076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41255 (local address 127.0.98.3:42043)
0619 16:41:43.332455 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:43.332473 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:43.332665 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:43.333349 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:43.333352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:43.333404 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:43.333557 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:43.333564 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.334752 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.334755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:43.335868 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:43.335875 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.336588 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.336591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:43.336706 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:43.336742 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 16:41:43.336837 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 16:41:43.336929 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"spinlock_wait_cycles":2432,"thread_start_us":90,"threads_started":1}
I20260619 16:41:43.337087 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:43.332259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42043 (local address 127.0.98.1:41255)
0619 16:41:43.332394 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:43.332400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:43.332560 (+ 160us) server_negotiation.cc:407] Connection header received
0619 16:41:43.332836 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:43.332842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:43.332956 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:43.333018 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:43.333745 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.334611 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.336000 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.336431 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.336583 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:43.336664 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:43.336854 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 16:41:43.336935 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:41:43.337357 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:43.332616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57545 (local address 127.0.98.2:43469)
0619 16:41:43.332760 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:43.332767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:43.332852 (+ 85us) server_negotiation.cc:407] Connection header received
0619 16:41:43.333008 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:43.333012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:43.333057 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:43.333313 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:41:43.333644 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.334177 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.335773 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.336466 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.336587 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:43.336676 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:43.337092 (+ 416us) server_negotiation.cc:299] Negotiation successful
0619 16:41:43.337186 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:41:43.337873 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:43.332277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43469 (local address 127.0.98.3:57545)
0619 16:41:43.332725 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:43.332741 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:43.332881 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:43.333327 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:43.333330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:43.333347 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:41:43.333515 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:43.333522 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.334298 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.334304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:43.335575 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:43.335587 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.336781 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.336787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:43.336923 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:43.336951 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 16:41:43.337575 (+ 624us) client_negotiation.cc:240] Negotiation successful
0619 16:41:43.337711 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":114,"threads_started":1}
I20260619 16:41:43.338114 8068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1044700b8be343978ca6bcfe592a01c8" candidate_uuid: "61292455bfa049e38d0cbda9e3d49542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a26293ac1ff74ce5a54b683d5f47954d" is_pre_election: true
I20260619 16:41:43.338341 8068 raft_consensus.cc:2454] T 1044700b8be343978ca6bcfe592a01c8 P a26293ac1ff74ce5a54b683d5f47954d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61292455bfa049e38d0cbda9e3d49542 in term 0.
I20260619 16:41:43.338485 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1044700b8be343978ca6bcfe592a01c8" candidate_uuid: "61292455bfa049e38d0cbda9e3d49542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b170ecc3958549ba8bf604b1b301c9f5" is_pre_election: true
I20260619 16:41:43.338748 8208 raft_consensus.cc:2454] T 1044700b8be343978ca6bcfe592a01c8 P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61292455bfa049e38d0cbda9e3d49542 in term 0.
I20260619 16:41:43.338827 8281 leader_election.cc:302] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61292455bfa049e38d0cbda9e3d49542, a26293ac1ff74ce5a54b683d5f47954d; no voters:
I20260619 16:41:43.339263 8428 raft_consensus.cc:2781] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:41:43.339362 8428 raft_consensus.cc:482] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:41:43.339419 8428 raft_consensus.cc:3037] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:43.340776 8428 raft_consensus.cc:504] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.341228 8428 leader_election.cc:288] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542 [CANDIDATE]: Term 1 election: Requested vote from peers a26293ac1ff74ce5a54b683d5f47954d (127.0.98.1:41255), b170ecc3958549ba8bf604b1b301c9f5 (127.0.98.2:43469)
I20260619 16:41:43.341683 8068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1044700b8be343978ca6bcfe592a01c8" candidate_uuid: "61292455bfa049e38d0cbda9e3d49542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a26293ac1ff74ce5a54b683d5f47954d"
I20260619 16:41:43.341714 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1044700b8be343978ca6bcfe592a01c8" candidate_uuid: "61292455bfa049e38d0cbda9e3d49542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b170ecc3958549ba8bf604b1b301c9f5"
I20260619 16:41:43.341852 8068 raft_consensus.cc:3037] T 1044700b8be343978ca6bcfe592a01c8 P a26293ac1ff74ce5a54b683d5f47954d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:43.341893 8208 raft_consensus.cc:3037] T 1044700b8be343978ca6bcfe592a01c8 P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:43.342890 8068 raft_consensus.cc:2454] T 1044700b8be343978ca6bcfe592a01c8 P a26293ac1ff74ce5a54b683d5f47954d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61292455bfa049e38d0cbda9e3d49542 in term 1.
I20260619 16:41:43.343255 8281 leader_election.cc:302] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61292455bfa049e38d0cbda9e3d49542, a26293ac1ff74ce5a54b683d5f47954d; no voters:
I20260619 16:41:43.343348 8208 raft_consensus.cc:2454] T 1044700b8be343978ca6bcfe592a01c8 P b170ecc3958549ba8bf604b1b301c9f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61292455bfa049e38d0cbda9e3d49542 in term 1.
I20260619 16:41:43.343479 8428 raft_consensus.cc:2781] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:43.343667 8428 raft_consensus.cc:686] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542 [term 1 LEADER]: Becoming Leader. State: Replica: 61292455bfa049e38d0cbda9e3d49542, State: Running, Role: LEADER
I20260619 16:41:43.344012 8428 consensus_queue.cc:237] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.346976 7911 catalog_manager.cc:5495] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61292455bfa049e38d0cbda9e3d49542 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "61292455bfa049e38d0cbda9e3d49542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } health_report { overall_health: HEALTHY } } }
Jun 19 16:41:43 dist-test-slave-03cd krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887301, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:41:43.397807 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:43.381750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39229 (local address 127.0.0.1:54476)
0619 16:41:43.382150 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:43.382175 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:43.382341 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:43.382638 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:43.382641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:43.383073 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:43.383158 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:43.383165 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.384025 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.384032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:43.384592 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:43.384599 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.385122 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.385125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:43.385213 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:43.391128 (+ 5915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:43.391146 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:43.393371 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:43.396019 (+ 2648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:43.396034 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:43.396044 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:43.396334 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:43.396644 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:43.396649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:43.396651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:43.396731 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:43.397143 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:43.397153 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:43.397262 (+ 109us) client_negotiation.cc:769] Sending connection context
0619 16:41:43.397378 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 16:41:43.397583 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":201,"threads_started":1}
I20260619 16:41:43.397807 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:43.381809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54476 (local address 127.0.98.62:39229)
0619 16:41:43.382111 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:43.382120 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:43.382272 (+ 152us) server_negotiation.cc:407] Connection header received
0619 16:41:43.382470 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:43.382473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:43.382513 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:43.382614 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:43.383288 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.383917 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.384712 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.385006 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.385115 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:43.393569 (+ 8454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:43.393573 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:43.393575 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:43.393597 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:43.395561 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:43.396465 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:43.396469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:43.396471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:43.396551 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:43.396817 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:43.396823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:43.396827 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:43.397026 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:43.397131 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:43.397450 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 16:41:43.397662 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":96,"server-negotiator.queue_time_us":217,"thread_start_us":144,"threads_started":1}
I20260619 16:41:43.403023 7911 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54476:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 16:41:43.404256 7911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:41:43.410459 8048 tablet_service.cc:1467] Processing CreateTablet for tablet 7872f02f87ed4f9b9d0e9b86a3b7ba4b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=501de41203fa487fb691698611c57d29]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:41:43.410629 8188 tablet_service.cc:1467] Processing CreateTablet for tablet 7872f02f87ed4f9b9d0e9b86a3b7ba4b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=501de41203fa487fb691698611c57d29]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:41:43.410955 8188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7872f02f87ed4f9b9d0e9b86a3b7ba4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:43.410983 8048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7872f02f87ed4f9b9d0e9b86a3b7ba4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:43.410799 8328 tablet_service.cc:1467] Processing CreateTablet for tablet 7872f02f87ed4f9b9d0e9b86a3b7ba4b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=501de41203fa487fb691698611c57d29]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:41:43.411388 8328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7872f02f87ed4f9b9d0e9b86a3b7ba4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:43.414245 8423 tablet_bootstrap.cc:492] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5: Bootstrap starting.
I20260619 16:41:43.415488 8423 tablet_bootstrap.cc:654] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:43.416278 8422 tablet_bootstrap.cc:492] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P a26293ac1ff74ce5a54b683d5f47954d: Bootstrap starting.
I20260619 16:41:43.417687 8422 tablet_bootstrap.cc:654] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P a26293ac1ff74ce5a54b683d5f47954d: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:43.419205 8423 tablet_bootstrap.cc:492] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5: No bootstrap required, opened a new log
I20260619 16:41:43.419288 8422 tablet_bootstrap.cc:492] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P a26293ac1ff74ce5a54b683d5f47954d: No bootstrap required, opened a new log
I20260619 16:41:43.419320 8423 ts_tablet_manager.cc:1397] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 16:41:43.419427 8422 ts_tablet_manager.cc:1397] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P a26293ac1ff74ce5a54b683d5f47954d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:41:43.419864 8423 raft_consensus.cc:348] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.420027 8423 raft_consensus.cc:374] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:43.420073 8423 raft_consensus.cc:729] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b170ecc3958549ba8bf604b1b301c9f5, State: Initialized, Role: FOLLOWER
I20260619 16:41:43.420131 8422 raft_consensus.cc:348] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P a26293ac1ff74ce5a54b683d5f47954d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.420274 8422 raft_consensus.cc:374] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P a26293ac1ff74ce5a54b683d5f47954d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:43.420274 8423 consensus_queue.cc:260] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.420326 8422 raft_consensus.cc:729] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P a26293ac1ff74ce5a54b683d5f47954d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a26293ac1ff74ce5a54b683d5f47954d, State: Initialized, Role: FOLLOWER
I20260619 16:41:43.420544 8442 tablet_replica.cc:439] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } } }
I20260619 16:41:43.420686 8442 tablet_replica.cc:442] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:41:43.420648 8422 consensus_queue.cc:260] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P a26293ac1ff74ce5a54b683d5f47954d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.421003 8422 ts_tablet_manager.cc:1428] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P a26293ac1ff74ce5a54b683d5f47954d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 16:41:43.421367 8429 tablet_replica.cc:439] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P a26293ac1ff74ce5a54b683d5f47954d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } } }
I20260619 16:41:43.421558 8429 tablet_replica.cc:442] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P a26293ac1ff74ce5a54b683d5f47954d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:41:43.422636 8424 tablet_bootstrap.cc:492] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P 61292455bfa049e38d0cbda9e3d49542: Bootstrap starting.
I20260619 16:41:43.423569 8423 ts_tablet_manager.cc:1428] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 16:41:43.423805 8424 tablet_bootstrap.cc:654] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P 61292455bfa049e38d0cbda9e3d49542: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:43.426265 8424 tablet_bootstrap.cc:492] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P 61292455bfa049e38d0cbda9e3d49542: No bootstrap required, opened a new log
I20260619 16:41:43.426375 8424 ts_tablet_manager.cc:1397] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P 61292455bfa049e38d0cbda9e3d49542: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260619 16:41:43.426784 8424 raft_consensus.cc:348] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.426930 8424 raft_consensus.cc:374] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:43.426976 8424 raft_consensus.cc:729] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61292455bfa049e38d0cbda9e3d49542, State: Initialized, Role: FOLLOWER
I20260619 16:41:43.427116 8424 consensus_queue.cc:260] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P 61292455bfa049e38d0cbda9e3d49542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.427419 8424 ts_tablet_manager.cc:1428] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P 61292455bfa049e38d0cbda9e3d49542: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 16:41:43.427374 8428 tablet_replica.cc:439] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P 61292455bfa049e38d0cbda9e3d49542: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } } }
I20260619 16:41:43.427507 8428 tablet_replica.cc:442] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P 61292455bfa049e38d0cbda9e3d49542: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:41:43.487104 8442 raft_consensus.cc:482] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:41:43.487280 8442 raft_consensus.cc:504] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.487841 8442 leader_election.cc:288] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a26293ac1ff74ce5a54b683d5f47954d (127.0.98.1:41255), 61292455bfa049e38d0cbda9e3d49542 (127.0.98.3:44575)
I20260619 16:41:43.488394 8068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7872f02f87ed4f9b9d0e9b86a3b7ba4b" candidate_uuid: "b170ecc3958549ba8bf604b1b301c9f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a26293ac1ff74ce5a54b683d5f47954d" is_pre_election: true
I20260619 16:41:43.488577 8068 raft_consensus.cc:2454] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P a26293ac1ff74ce5a54b683d5f47954d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b170ecc3958549ba8bf604b1b301c9f5 in term 0.
I20260619 16:41:43.488507 8348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7872f02f87ed4f9b9d0e9b86a3b7ba4b" candidate_uuid: "b170ecc3958549ba8bf604b1b301c9f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61292455bfa049e38d0cbda9e3d49542" is_pre_election: true
I20260619 16:41:43.488811 8348 raft_consensus.cc:2454] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b170ecc3958549ba8bf604b1b301c9f5 in term 0.
I20260619 16:41:43.488953 8141 leader_election.cc:302] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a26293ac1ff74ce5a54b683d5f47954d, b170ecc3958549ba8bf604b1b301c9f5; no voters:
I20260619 16:41:43.489171 8442 raft_consensus.cc:2781] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:41:43.489219 8442 raft_consensus.cc:482] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:41:43.489298 8442 raft_consensus.cc:3037] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:43.490427 8442 raft_consensus.cc:504] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.490870 8442 leader_election.cc:288] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5 [CANDIDATE]: Term 1 election: Requested vote from peers a26293ac1ff74ce5a54b683d5f47954d (127.0.98.1:41255), 61292455bfa049e38d0cbda9e3d49542 (127.0.98.3:44575)
I20260619 16:41:43.491299 8348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7872f02f87ed4f9b9d0e9b86a3b7ba4b" candidate_uuid: "b170ecc3958549ba8bf604b1b301c9f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61292455bfa049e38d0cbda9e3d49542"
I20260619 16:41:43.491395 8068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7872f02f87ed4f9b9d0e9b86a3b7ba4b" candidate_uuid: "b170ecc3958549ba8bf604b1b301c9f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a26293ac1ff74ce5a54b683d5f47954d"
I20260619 16:41:43.491472 8348 raft_consensus.cc:3037] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:43.491508 8068 raft_consensus.cc:3037] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P a26293ac1ff74ce5a54b683d5f47954d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:43.492733 8068 raft_consensus.cc:2454] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P a26293ac1ff74ce5a54b683d5f47954d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b170ecc3958549ba8bf604b1b301c9f5 in term 1.
I20260619 16:41:43.492733 8348 raft_consensus.cc:2454] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P 61292455bfa049e38d0cbda9e3d49542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b170ecc3958549ba8bf604b1b301c9f5 in term 1.
I20260619 16:41:43.493113 8141 leader_election.cc:302] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a26293ac1ff74ce5a54b683d5f47954d, b170ecc3958549ba8bf604b1b301c9f5; no voters:
I20260619 16:41:43.493366 8442 raft_consensus.cc:2781] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:43.493456 8442 raft_consensus.cc:686] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5 [term 1 LEADER]: Becoming Leader. State: Replica: b170ecc3958549ba8bf604b1b301c9f5, State: Running, Role: LEADER
I20260619 16:41:43.493624 8442 consensus_queue.cc:237] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.494462 8433 tablet_replica.cc:439] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5: TxnStatusTablet state changed. Reason: New leader b170ecc3958549ba8bf604b1b301c9f5. Latest consensus state: current_term: 1 leader_uuid: "b170ecc3958549ba8bf604b1b301c9f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } } }
I20260619 16:41:43.494581 8433 tablet_replica.cc:442] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:41:43.494882 8467 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 16:41:43.495055 8467 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 16:41:43.495607 7911 catalog_manager.cc:5495] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b170ecc3958549ba8bf604b1b301c9f5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b170ecc3958549ba8bf604b1b301c9f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } health_report { overall_health: UNKNOWN } } }
I20260619 16:41:43.515394 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:43.510812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37480 (local address 127.0.98.2:43469)
0619 16:41:43.510960 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:43.510964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:43.510985 (+ 21us) server_negotiation.cc:407] Connection header received
0619 16:41:43.511121 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:43.511125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:43.511217 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:43.511329 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:41:43.511703 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.512651 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.514008 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.514336 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.514483 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:43.514843 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:41:43.514956 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:41:43.515033 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:43.515192 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 16:41:43.515265 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 16:41:43.515434 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:43.510678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43469 (local address 127.0.0.1:37480)
0619 16:41:43.510829 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:43.510847 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:43.510965 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:43.511374 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:43.511377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:43.511397 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:41:43.511534 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:43.511542 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.512822 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.512825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:43.513845 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:43.513857 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.514479 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.514482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:43.514579 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:43.514642 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:41:43.515065 (+ 423us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:41:43.515079 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 16:41:43.515160 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 16:41:43.515257 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 16:41:43.519114 8348 raft_consensus.cc:1261] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P 61292455bfa049e38d0cbda9e3d49542 [term 1 FOLLOWER]: Refusing update from remote peer b170ecc3958549ba8bf604b1b301c9f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:43.519070 8068 raft_consensus.cc:1261] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P a26293ac1ff74ce5a54b683d5f47954d [term 1 FOLLOWER]: Refusing update from remote peer b170ecc3958549ba8bf604b1b301c9f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:43.519644 8442 consensus_queue.cc:1035] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:43.519870 8433 consensus_queue.cc:1035] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:43.521436 8428 tablet_replica.cc:439] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P 61292455bfa049e38d0cbda9e3d49542: TxnStatusTablet state changed. Reason: New leader b170ecc3958549ba8bf604b1b301c9f5. Latest consensus state: current_term: 1 leader_uuid: "b170ecc3958549ba8bf604b1b301c9f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } } }
I20260619 16:41:43.521603 8428 tablet_replica.cc:442] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P 61292455bfa049e38d0cbda9e3d49542: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:41:43.522715 8429 tablet_replica.cc:439] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P a26293ac1ff74ce5a54b683d5f47954d: TxnStatusTablet state changed. Reason: New leader b170ecc3958549ba8bf604b1b301c9f5. Latest consensus state: current_term: 1 leader_uuid: "b170ecc3958549ba8bf604b1b301c9f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } } }
I20260619 16:41:43.522877 8429 tablet_replica.cc:442] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P a26293ac1ff74ce5a54b683d5f47954d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:41:43.524014 8433 tablet_replica.cc:439] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b170ecc3958549ba8bf604b1b301c9f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } } }
I20260619 16:41:43.524163 8433 tablet_replica.cc:442] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:41:43.524145 8436 mvcc.cc:205] Tried to move back new op lower bound from 7298610395207675904 to 7298610395111575552. Current Snapshot: MvccSnapshot[applied={T|T < 7298610395207675904}]
I20260619 16:41:43.525000 8429 tablet_replica.cc:439] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P a26293ac1ff74ce5a54b683d5f47954d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b170ecc3958549ba8bf604b1b301c9f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } } }
I20260619 16:41:43.525118 8429 tablet_replica.cc:442] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P a26293ac1ff74ce5a54b683d5f47954d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:41:43.529763 8430 tablet_replica.cc:439] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b170ecc3958549ba8bf604b1b301c9f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } } }
I20260619 16:41:43.529942 8430 tablet_replica.cc:442] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P b170ecc3958549ba8bf604b1b301c9f5: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:41:43.530788 8437 mvcc.cc:205] Tried to move back new op lower bound from 7298610395207675904 to 7298610395111575552. Current Snapshot: MvccSnapshot[applied={T|T < 7298610395207675904}]
I20260619 16:41:43.531025 8428 tablet_replica.cc:439] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P 61292455bfa049e38d0cbda9e3d49542: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b170ecc3958549ba8bf604b1b301c9f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } } }
I20260619 16:41:43.531147 8428 tablet_replica.cc:442] T 7872f02f87ed4f9b9d0e9b86a3b7ba4b P 61292455bfa049e38d0cbda9e3d49542: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:41:43.537389 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:43.532417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44198 (local address 127.0.98.3:44575)
0619 16:41:43.532564 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:43.532569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:43.532584 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:41:43.532710 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:43.532713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:43.532789 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:43.532922 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:41:43.533299 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.534032 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.535357 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.535649 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.535852 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:43.536175 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:41:43.536824 (+ 649us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:41:43.536932 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:43.537062 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 16:41:43.537133 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:41:43.551412 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:43.547137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43469 (local address 127.0.98.3:58401)
0619 16:41:43.547578 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:43.547593 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:43.547840 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:43.548131 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:43.548134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:43.548141 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:41:43.548287 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:43.548292 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.549157 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.549163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:43.549941 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:43.549950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.550523 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.550529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:43.550618 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:43.550678 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:41:43.551017 (+ 339us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:41:43.551027 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 16:41:43.551115 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 16:41:43.551206 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":143,"threads_started":1}
I20260619 16:41:43.551385 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:43.547271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58401 (local address 127.0.98.2:43469)
0619 16:41:43.547427 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:43.547434 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:43.547755 (+ 321us) server_negotiation.cc:407] Connection header received
0619 16:41:43.547964 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:43.547972 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:43.548026 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:43.548110 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:41:43.548468 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.549035 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.550071 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.550363 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.550481 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:43.550835 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:41:43.550934 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:41:43.551015 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:43.551127 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 16:41:43.551205 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:41:43.560370 8068 raft_consensus.cc:1261] T 1044700b8be343978ca6bcfe592a01c8 P a26293ac1ff74ce5a54b683d5f47954d [term 1 FOLLOWER]: Refusing update from remote peer 61292455bfa049e38d0cbda9e3d49542: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:43.560545 8208 raft_consensus.cc:1261] T 1044700b8be343978ca6bcfe592a01c8 P b170ecc3958549ba8bf604b1b301c9f5 [term 1 FOLLOWER]: Refusing update from remote peer 61292455bfa049e38d0cbda9e3d49542: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:43.560967 8428 consensus_queue.cc:1035] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:43.561496 8454 consensus_queue.cc:1035] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:43.593703 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:43.588946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44575 (local address 127.0.98.2:53679)
0619 16:41:43.589382 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:43.589400 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:43.589531 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:43.589861 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:43.589864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:43.589879 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:41:43.590057 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:43.590065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.590933 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.590937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:43.592000 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:43.592011 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.592585 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.592589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:43.592701 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:43.592760 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:41:43.593326 (+ 566us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:41:43.593334 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 16:41:43.593427 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 16:41:43.593545 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":185,"threads_started":1}
I20260619 16:41:43.593747 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:43.589368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53679 (local address 127.0.98.3:44575)
0619 16:41:43.589503 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:43.589508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:43.589522 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:41:43.589637 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:43.589640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:43.589707 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:43.589806 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:41:43.590181 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.590789 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.592165 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.592448 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.592576 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:43.593107 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:41:43.593222 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:41:43.593306 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:43.593451 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 16:41:43.593512 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:41:43.626850 7910 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:41:43.627030 7910 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:41:43.629871 7910 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81e280c69e54404f838b915627e155a8: Sending DeleteTablet for 3 replicas of tablet 1044700b8be343978ca6bcfe592a01c8
I20260619 16:41:43.630746 8048 tablet_service.cc:1514] Processing DeleteTablet for tablet 1044700b8be343978ca6bcfe592a01c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50444
I20260619 16:41:43.630952 8188 tablet_service.cc:1514] Processing DeleteTablet for tablet 1044700b8be343978ca6bcfe592a01c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37470
I20260619 16:41:43.631071 8447 tablet_replica.cc:330] T 1044700b8be343978ca6bcfe592a01c8 P a26293ac1ff74ce5a54b683d5f47954d: stopping tablet replica
I20260619 16:41:43.631198 8447 raft_consensus.cc:2229] T 1044700b8be343978ca6bcfe592a01c8 P a26293ac1ff74ce5a54b683d5f47954d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:41:43.631349 8447 raft_consensus.cc:2258] T 1044700b8be343978ca6bcfe592a01c8 P a26293ac1ff74ce5a54b683d5f47954d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:43.631572 8448 tablet_replica.cc:330] T 1044700b8be343978ca6bcfe592a01c8 P b170ecc3958549ba8bf604b1b301c9f5: stopping tablet replica
I20260619 16:41:43.631757 8448 raft_consensus.cc:2229] T 1044700b8be343978ca6bcfe592a01c8 P b170ecc3958549ba8bf604b1b301c9f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:41:43.631899 8448 raft_consensus.cc:2258] T 1044700b8be343978ca6bcfe592a01c8 P b170ecc3958549ba8bf604b1b301c9f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:43.632105 8328 tablet_service.cc:1514] Processing DeleteTablet for tablet 1044700b8be343978ca6bcfe592a01c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44182
I20260619 16:41:43.632310 8446 tablet_replica.cc:330] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542: stopping tablet replica
I20260619 16:41:43.632439 8446 raft_consensus.cc:2229] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:41:43.632805 8446 raft_consensus.cc:2258] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:43.633836 8448 ts_tablet_manager.cc:1905] T 1044700b8be343978ca6bcfe592a01c8 P b170ecc3958549ba8bf604b1b301c9f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:43.633912 8446 ts_tablet_manager.cc:1905] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:43.637138 8448 ts_tablet_manager.cc:1918] T 1044700b8be343978ca6bcfe592a01c8 P b170ecc3958549ba8bf604b1b301c9f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:41:43.637247 8448 log.cc:1196] T 1044700b8be343978ca6bcfe592a01c8 P b170ecc3958549ba8bf604b1b301c9f5: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-1/wal/wals/1044700b8be343978ca6bcfe592a01c8
I20260619 16:41:43.637542 8448 ts_tablet_manager.cc:1939] T 1044700b8be343978ca6bcfe592a01c8 P b170ecc3958549ba8bf604b1b301c9f5: Deleting consensus metadata
I20260619 16:41:43.638217 8446 ts_tablet_manager.cc:1918] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:41:43.638398 8446 log.cc:1196] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-2/wal/wals/1044700b8be343978ca6bcfe592a01c8
I20260619 16:41:43.638391 7895 catalog_manager.cc:4841] TS b170ecc3958549ba8bf604b1b301c9f5 (127.0.98.2:43469): tablet 1044700b8be343978ca6bcfe592a01c8 (table test-table [id=134dd24f47bc41548e41d8fdfd83ce3c]) successfully deleted
I20260619 16:41:43.638804 8446 ts_tablet_manager.cc:1939] T 1044700b8be343978ca6bcfe592a01c8 P 61292455bfa049e38d0cbda9e3d49542: Deleting consensus metadata
I20260619 16:41:43.639330 8447 ts_tablet_manager.cc:1905] T 1044700b8be343978ca6bcfe592a01c8 P a26293ac1ff74ce5a54b683d5f47954d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:43.642277 8447 ts_tablet_manager.cc:1918] T 1044700b8be343978ca6bcfe592a01c8 P a26293ac1ff74ce5a54b683d5f47954d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:41:43.642372 8447 log.cc:1196] T 1044700b8be343978ca6bcfe592a01c8 P a26293ac1ff74ce5a54b683d5f47954d: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-0/wal/wals/1044700b8be343978ca6bcfe592a01c8
I20260619 16:41:43.642652 8447 ts_tablet_manager.cc:1939] T 1044700b8be343978ca6bcfe592a01c8 P a26293ac1ff74ce5a54b683d5f47954d: Deleting consensus metadata
I20260619 16:41:43.643507 7897 catalog_manager.cc:4841] TS a26293ac1ff74ce5a54b683d5f47954d (127.0.98.1:41255): tablet 1044700b8be343978ca6bcfe592a01c8 (table test-table [id=134dd24f47bc41548e41d8fdfd83ce3c]) successfully deleted
I20260619 16:41:43.643513 7895 catalog_manager.cc:4841] TS 61292455bfa049e38d0cbda9e3d49542 (127.0.98.3:44575): tablet 1044700b8be343978ca6bcfe592a01c8 (table test-table [id=134dd24f47bc41548e41d8fdfd83ce3c]) successfully deleted
Jun 19 16:41:43 dist-test-slave-03cd krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:41:43 dist-test-slave-03cd krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:43 dist-test-slave-03cd krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 16:41:43.681869 8461 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:43.672056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54490 (local address 127.0.98.62:39229)
0619 16:41:43.672265 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:43.672272 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:43.672822 (+ 550us) server_negotiation.cc:407] Connection header received
0619 16:41:43.673000 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:43.673005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:43.673092 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:43.673209 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:43.674383 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.675360 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.676812 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.677209 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.677354 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:43.681689 (+ 4335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54490: BlockingRecv error: recv got EOF from 127.0.0.1:54490 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 19 16:41:43 dist-test-slave-03cd krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:41:43.696776 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:43.684533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54496 (local address 127.0.98.62:39229)
0619 16:41:43.684693 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:43.684699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:43.684814 (+ 115us) server_negotiation.cc:407] Connection header received
0619 16:41:43.684956 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:43.684960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:43.685026 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:43.685144 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:43.685901 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.686909 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.688206 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.688633 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.688800 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:43.692303 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:43.692309 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:43.692313 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:43.692365 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:43.694349 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:43.694960 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:43.694965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:43.694967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:43.695085 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:43.695532 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:43.695535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:43.695537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:43.695884 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:43.696052 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:43.696374 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 16:41:43.696556 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 16:41:43.700899 7910 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:41:43.701543 7910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:41:43.707784 8048 tablet_service.cc:1467] Processing CreateTablet for tablet 92e2914ebdb34325bdcaed8a4d16ef67 (DEFAULT_TABLE table=test-table [id=30bec7cfb3d34cbf8d35215558089e64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:43.707828 8328 tablet_service.cc:1467] Processing CreateTablet for tablet 92e2914ebdb34325bdcaed8a4d16ef67 (DEFAULT_TABLE table=test-table [id=30bec7cfb3d34cbf8d35215558089e64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:43.708216 8048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92e2914ebdb34325bdcaed8a4d16ef67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:43.708222 8328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92e2914ebdb34325bdcaed8a4d16ef67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:43.710450 8424 tablet_bootstrap.cc:492] T 92e2914ebdb34325bdcaed8a4d16ef67 P 61292455bfa049e38d0cbda9e3d49542: Bootstrap starting.
I20260619 16:41:43.710753 8188 tablet_service.cc:1467] Processing CreateTablet for tablet 92e2914ebdb34325bdcaed8a4d16ef67 (DEFAULT_TABLE table=test-table [id=30bec7cfb3d34cbf8d35215558089e64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:43.711048 8188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92e2914ebdb34325bdcaed8a4d16ef67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:43.711514 8424 tablet_bootstrap.cc:654] T 92e2914ebdb34325bdcaed8a4d16ef67 P 61292455bfa049e38d0cbda9e3d49542: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:43.713001 8424 tablet_bootstrap.cc:492] T 92e2914ebdb34325bdcaed8a4d16ef67 P 61292455bfa049e38d0cbda9e3d49542: No bootstrap required, opened a new log
I20260619 16:41:43.713079 8424 ts_tablet_manager.cc:1397] T 92e2914ebdb34325bdcaed8a4d16ef67 P 61292455bfa049e38d0cbda9e3d49542: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 16:41:43.713574 8424 raft_consensus.cc:348] T 92e2914ebdb34325bdcaed8a4d16ef67 P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.713796 8423 tablet_bootstrap.cc:492] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5: Bootstrap starting.
I20260619 16:41:43.713821 8424 raft_consensus.cc:374] T 92e2914ebdb34325bdcaed8a4d16ef67 P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:43.713883 8424 raft_consensus.cc:729] T 92e2914ebdb34325bdcaed8a4d16ef67 P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61292455bfa049e38d0cbda9e3d49542, State: Initialized, Role: FOLLOWER
I20260619 16:41:43.714004 8424 consensus_queue.cc:260] T 92e2914ebdb34325bdcaed8a4d16ef67 P 61292455bfa049e38d0cbda9e3d49542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.714100 8422 tablet_bootstrap.cc:492] T 92e2914ebdb34325bdcaed8a4d16ef67 P a26293ac1ff74ce5a54b683d5f47954d: Bootstrap starting.
I20260619 16:41:43.714259 8424 ts_tablet_manager.cc:1428] T 92e2914ebdb34325bdcaed8a4d16ef67 P 61292455bfa049e38d0cbda9e3d49542: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 16:41:43.715037 8422 tablet_bootstrap.cc:654] T 92e2914ebdb34325bdcaed8a4d16ef67 P a26293ac1ff74ce5a54b683d5f47954d: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:43.715080 8423 tablet_bootstrap.cc:654] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:43.716533 8422 tablet_bootstrap.cc:492] T 92e2914ebdb34325bdcaed8a4d16ef67 P a26293ac1ff74ce5a54b683d5f47954d: No bootstrap required, opened a new log
I20260619 16:41:43.716624 8422 ts_tablet_manager.cc:1397] T 92e2914ebdb34325bdcaed8a4d16ef67 P a26293ac1ff74ce5a54b683d5f47954d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 16:41:43.717155 8422 raft_consensus.cc:348] T 92e2914ebdb34325bdcaed8a4d16ef67 P a26293ac1ff74ce5a54b683d5f47954d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.717321 8422 raft_consensus.cc:374] T 92e2914ebdb34325bdcaed8a4d16ef67 P a26293ac1ff74ce5a54b683d5f47954d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:43.717353 8422 raft_consensus.cc:729] T 92e2914ebdb34325bdcaed8a4d16ef67 P a26293ac1ff74ce5a54b683d5f47954d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a26293ac1ff74ce5a54b683d5f47954d, State: Initialized, Role: FOLLOWER
I20260619 16:41:43.717489 8422 consensus_queue.cc:260] T 92e2914ebdb34325bdcaed8a4d16ef67 P a26293ac1ff74ce5a54b683d5f47954d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.717811 8423 tablet_bootstrap.cc:492] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5: No bootstrap required, opened a new log
I20260619 16:41:43.717900 8423 ts_tablet_manager.cc:1397] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 16:41:43.717973 8422 ts_tablet_manager.cc:1428] T 92e2914ebdb34325bdcaed8a4d16ef67 P a26293ac1ff74ce5a54b683d5f47954d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 16:41:43.718304 8423 raft_consensus.cc:348] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.718451 8423 raft_consensus.cc:374] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:43.718492 8423 raft_consensus.cc:729] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b170ecc3958549ba8bf604b1b301c9f5, State: Initialized, Role: FOLLOWER
I20260619 16:41:43.718645 8423 consensus_queue.cc:260] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.718940 8423 ts_tablet_manager.cc:1428] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:41:43.719482 8442 raft_consensus.cc:482] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:41:43.719571 8442 raft_consensus.cc:504] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.720011 8442 leader_election.cc:288] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a26293ac1ff74ce5a54b683d5f47954d (127.0.98.1:41255), 61292455bfa049e38d0cbda9e3d49542 (127.0.98.3:44575)
I20260619 16:41:43.720502 8068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92e2914ebdb34325bdcaed8a4d16ef67" candidate_uuid: "b170ecc3958549ba8bf604b1b301c9f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a26293ac1ff74ce5a54b683d5f47954d" is_pre_election: true
I20260619 16:41:43.720635 8068 raft_consensus.cc:2454] T 92e2914ebdb34325bdcaed8a4d16ef67 P a26293ac1ff74ce5a54b683d5f47954d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b170ecc3958549ba8bf604b1b301c9f5 in term 0.
I20260619 16:41:43.721131 8141 leader_election.cc:302] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a26293ac1ff74ce5a54b683d5f47954d, b170ecc3958549ba8bf604b1b301c9f5; no voters:
I20260619 16:41:43.721736 8348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92e2914ebdb34325bdcaed8a4d16ef67" candidate_uuid: "b170ecc3958549ba8bf604b1b301c9f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61292455bfa049e38d0cbda9e3d49542" is_pre_election: true
I20260619 16:41:43.721964 8442 raft_consensus.cc:2781] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:41:43.721964 8348 raft_consensus.cc:2454] T 92e2914ebdb34325bdcaed8a4d16ef67 P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b170ecc3958549ba8bf604b1b301c9f5 in term 0.
I20260619 16:41:43.722041 8442 raft_consensus.cc:482] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:41:43.722083 8442 raft_consensus.cc:3037] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:43.723037 8442 raft_consensus.cc:504] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.723413 8442 leader_election.cc:288] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5 [CANDIDATE]: Term 1 election: Requested vote from peers a26293ac1ff74ce5a54b683d5f47954d (127.0.98.1:41255), 61292455bfa049e38d0cbda9e3d49542 (127.0.98.3:44575)
I20260619 16:41:43.723860 8348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92e2914ebdb34325bdcaed8a4d16ef67" candidate_uuid: "b170ecc3958549ba8bf604b1b301c9f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61292455bfa049e38d0cbda9e3d49542"
I20260619 16:41:43.723932 8068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92e2914ebdb34325bdcaed8a4d16ef67" candidate_uuid: "b170ecc3958549ba8bf604b1b301c9f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a26293ac1ff74ce5a54b683d5f47954d"
I20260619 16:41:43.724002 8348 raft_consensus.cc:3037] T 92e2914ebdb34325bdcaed8a4d16ef67 P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:43.724089 8068 raft_consensus.cc:3037] T 92e2914ebdb34325bdcaed8a4d16ef67 P a26293ac1ff74ce5a54b683d5f47954d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:43.725029 8348 raft_consensus.cc:2454] T 92e2914ebdb34325bdcaed8a4d16ef67 P 61292455bfa049e38d0cbda9e3d49542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b170ecc3958549ba8bf604b1b301c9f5 in term 1.
I20260619 16:41:43.725337 8139 leader_election.cc:302] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61292455bfa049e38d0cbda9e3d49542, b170ecc3958549ba8bf604b1b301c9f5; no voters:
I20260619 16:41:43.725499 8068 raft_consensus.cc:2454] T 92e2914ebdb34325bdcaed8a4d16ef67 P a26293ac1ff74ce5a54b683d5f47954d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b170ecc3958549ba8bf604b1b301c9f5 in term 1.
I20260619 16:41:43.725524 8442 raft_consensus.cc:2781] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:43.725634 8442 raft_consensus.cc:686] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5 [term 1 LEADER]: Becoming Leader. State: Replica: b170ecc3958549ba8bf604b1b301c9f5, State: Running, Role: LEADER
I20260619 16:41:43.725783 8442 consensus_queue.cc:237] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.727854 7911 catalog_manager.cc:5495] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b170ecc3958549ba8bf604b1b301c9f5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b170ecc3958549ba8bf604b1b301c9f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } health_report { overall_health: UNKNOWN } } }
I20260619 16:41:43.739831 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:43.735466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37496 (local address 127.0.98.2:43469)
0619 16:41:43.735635 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:43.735640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:43.735695 (+ 55us) server_negotiation.cc:407] Connection header received
0619 16:41:43.735765 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:43.735768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:43.735836 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:43.735961 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:41:43.736298 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.737121 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.738366 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.738763 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.738883 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:43.739192 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:41:43.739316 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:41:43.739390 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:43.739517 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 16:41:43.739593 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:41:43.742563 8348 raft_consensus.cc:1261] T 92e2914ebdb34325bdcaed8a4d16ef67 P 61292455bfa049e38d0cbda9e3d49542 [term 1 FOLLOWER]: Refusing update from remote peer b170ecc3958549ba8bf604b1b301c9f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:43.742551 8068 raft_consensus.cc:1261] T 92e2914ebdb34325bdcaed8a4d16ef67 P a26293ac1ff74ce5a54b683d5f47954d [term 1 FOLLOWER]: Refusing update from remote peer b170ecc3958549ba8bf604b1b301c9f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:43.743119 8433 consensus_queue.cc:1035] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:43.743335 8442 consensus_queue.cc:1035] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:43.755177 7909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:41:43.755366 7909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:41:43.759341 7909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81e280c69e54404f838b915627e155a8: Sending DeleteTablet for 3 replicas of tablet 92e2914ebdb34325bdcaed8a4d16ef67
I20260619 16:41:43.760215 8328 tablet_service.cc:1514] Processing DeleteTablet for tablet 92e2914ebdb34325bdcaed8a4d16ef67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44182
I20260619 16:41:43.760499 8446 tablet_replica.cc:330] T 92e2914ebdb34325bdcaed8a4d16ef67 P 61292455bfa049e38d0cbda9e3d49542: stopping tablet replica
I20260619 16:41:43.760650 8446 raft_consensus.cc:2229] T 92e2914ebdb34325bdcaed8a4d16ef67 P 61292455bfa049e38d0cbda9e3d49542 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:41:43.760663 8048 tablet_service.cc:1514] Processing DeleteTablet for tablet 92e2914ebdb34325bdcaed8a4d16ef67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50444
I20260619 16:41:43.760824 8446 raft_consensus.cc:2258] T 92e2914ebdb34325bdcaed8a4d16ef67 P 61292455bfa049e38d0cbda9e3d49542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:43.760905 8447 tablet_replica.cc:330] T 92e2914ebdb34325bdcaed8a4d16ef67 P a26293ac1ff74ce5a54b683d5f47954d: stopping tablet replica
I20260619 16:41:43.760826 7909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 16:41:43.761034 8447 raft_consensus.cc:2229] T 92e2914ebdb34325bdcaed8a4d16ef67 P a26293ac1ff74ce5a54b683d5f47954d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:41:43.761175 8447 raft_consensus.cc:2258] T 92e2914ebdb34325bdcaed8a4d16ef67 P a26293ac1ff74ce5a54b683d5f47954d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 16:41:43.761312 7909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:41:43.761404 8188 tablet_service.cc:1514] Processing DeleteTablet for tablet 92e2914ebdb34325bdcaed8a4d16ef67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37470
I20260619 16:41:43.761624 8448 tablet_replica.cc:330] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5: stopping tablet replica
I20260619 16:41:43.761755 8448 raft_consensus.cc:2229] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:41:43.762012 8448 raft_consensus.cc:2258] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:43.764139 8447 ts_tablet_manager.cc:1905] T 92e2914ebdb34325bdcaed8a4d16ef67 P a26293ac1ff74ce5a54b683d5f47954d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:43.766739 8447 ts_tablet_manager.cc:1918] T 92e2914ebdb34325bdcaed8a4d16ef67 P a26293ac1ff74ce5a54b683d5f47954d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:41:43.766826 8447 log.cc:1196] T 92e2914ebdb34325bdcaed8a4d16ef67 P a26293ac1ff74ce5a54b683d5f47954d: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-0/wal/wals/92e2914ebdb34325bdcaed8a4d16ef67
I20260619 16:41:43.767092 8447 ts_tablet_manager.cc:1939] T 92e2914ebdb34325bdcaed8a4d16ef67 P a26293ac1ff74ce5a54b683d5f47954d: Deleting consensus metadata
I20260619 16:41:43.767922 8448 ts_tablet_manager.cc:1905] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:43.768822 8328 tablet_service.cc:1467] Processing CreateTablet for tablet cf985a6b6d2c43c5ad29126e47ab5c97 (DEFAULT_TABLE table=test-table [id=82753187307d4e428461679b44d0582a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:43.769318 8328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf985a6b6d2c43c5ad29126e47ab5c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:43.771353 8448 ts_tablet_manager.cc:1918] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:41:43.771451 8448 log.cc:1196] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-1/wal/wals/92e2914ebdb34325bdcaed8a4d16ef67
I20260619 16:41:43.772993 8188 tablet_service.cc:1467] Processing CreateTablet for tablet cf985a6b6d2c43c5ad29126e47ab5c97 (DEFAULT_TABLE table=test-table [id=82753187307d4e428461679b44d0582a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:43.773427 8188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf985a6b6d2c43c5ad29126e47ab5c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:43.773675 8048 tablet_service.cc:1467] Processing CreateTablet for tablet cf985a6b6d2c43c5ad29126e47ab5c97 (DEFAULT_TABLE table=test-table [id=82753187307d4e428461679b44d0582a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:41:43.774060 8048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf985a6b6d2c43c5ad29126e47ab5c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:43.776867 8422 tablet_bootstrap.cc:492] T cf985a6b6d2c43c5ad29126e47ab5c97 P a26293ac1ff74ce5a54b683d5f47954d: Bootstrap starting.
I20260619 16:41:43.777474 8423 tablet_bootstrap.cc:492] T cf985a6b6d2c43c5ad29126e47ab5c97 P b170ecc3958549ba8bf604b1b301c9f5: Bootstrap starting.
I20260619 16:41:43.777954 8422 tablet_bootstrap.cc:654] T cf985a6b6d2c43c5ad29126e47ab5c97 P a26293ac1ff74ce5a54b683d5f47954d: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:43.778705 8423 tablet_bootstrap.cc:654] T cf985a6b6d2c43c5ad29126e47ab5c97 P b170ecc3958549ba8bf604b1b301c9f5: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:43.779243 8448 ts_tablet_manager.cc:1939] T 92e2914ebdb34325bdcaed8a4d16ef67 P b170ecc3958549ba8bf604b1b301c9f5: Deleting consensus metadata
I20260619 16:41:43.780135 7897 catalog_manager.cc:4841] TS a26293ac1ff74ce5a54b683d5f47954d (127.0.98.1:41255): tablet 92e2914ebdb34325bdcaed8a4d16ef67 (table test-table [id=30bec7cfb3d34cbf8d35215558089e64]) successfully deleted
I20260619 16:41:43.780577 7895 catalog_manager.cc:4841] TS b170ecc3958549ba8bf604b1b301c9f5 (127.0.98.2:43469): tablet 92e2914ebdb34325bdcaed8a4d16ef67 (table test-table [id=30bec7cfb3d34cbf8d35215558089e64]) successfully deleted
I20260619 16:41:43.780881 8423 tablet_bootstrap.cc:492] T cf985a6b6d2c43c5ad29126e47ab5c97 P b170ecc3958549ba8bf604b1b301c9f5: No bootstrap required, opened a new log
I20260619 16:41:43.780972 8423 ts_tablet_manager.cc:1397] T cf985a6b6d2c43c5ad29126e47ab5c97 P b170ecc3958549ba8bf604b1b301c9f5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 16:41:43.781416 8423 raft_consensus.cc:348] T cf985a6b6d2c43c5ad29126e47ab5c97 P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.781736 8423 raft_consensus.cc:374] T cf985a6b6d2c43c5ad29126e47ab5c97 P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:43.781816 8423 raft_consensus.cc:729] T cf985a6b6d2c43c5ad29126e47ab5c97 P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b170ecc3958549ba8bf604b1b301c9f5, State: Initialized, Role: FOLLOWER
I20260619 16:41:43.782035 8423 consensus_queue.cc:260] T cf985a6b6d2c43c5ad29126e47ab5c97 P b170ecc3958549ba8bf604b1b301c9f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.782358 8424 tablet_bootstrap.cc:492] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542: Bootstrap starting.
I20260619 16:41:43.782389 8422 tablet_bootstrap.cc:492] T cf985a6b6d2c43c5ad29126e47ab5c97 P a26293ac1ff74ce5a54b683d5f47954d: No bootstrap required, opened a new log
I20260619 16:41:43.782459 8423 ts_tablet_manager.cc:1428] T cf985a6b6d2c43c5ad29126e47ab5c97 P b170ecc3958549ba8bf604b1b301c9f5: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 16:41:43.782506 8422 ts_tablet_manager.cc:1397] T cf985a6b6d2c43c5ad29126e47ab5c97 P a26293ac1ff74ce5a54b683d5f47954d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260619 16:41:43.783257 8422 raft_consensus.cc:348] T cf985a6b6d2c43c5ad29126e47ab5c97 P a26293ac1ff74ce5a54b683d5f47954d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.783404 8422 raft_consensus.cc:374] T cf985a6b6d2c43c5ad29126e47ab5c97 P a26293ac1ff74ce5a54b683d5f47954d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:43.783432 8422 raft_consensus.cc:729] T cf985a6b6d2c43c5ad29126e47ab5c97 P a26293ac1ff74ce5a54b683d5f47954d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a26293ac1ff74ce5a54b683d5f47954d, State: Initialized, Role: FOLLOWER
I20260619 16:41:43.783557 8422 consensus_queue.cc:260] T cf985a6b6d2c43c5ad29126e47ab5c97 P a26293ac1ff74ce5a54b683d5f47954d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.784147 8424 tablet_bootstrap.cc:654] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:43.784442 8422 ts_tablet_manager.cc:1428] T cf985a6b6d2c43c5ad29126e47ab5c97 P a26293ac1ff74ce5a54b683d5f47954d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260619 16:41:43.785950 8446 ts_tablet_manager.cc:1905] T 92e2914ebdb34325bdcaed8a4d16ef67 P 61292455bfa049e38d0cbda9e3d49542: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:43.789382 8446 ts_tablet_manager.cc:1918] T 92e2914ebdb34325bdcaed8a4d16ef67 P 61292455bfa049e38d0cbda9e3d49542: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:41:43.789534 8446 log.cc:1196] T 92e2914ebdb34325bdcaed8a4d16ef67 P 61292455bfa049e38d0cbda9e3d49542: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-2/wal/wals/92e2914ebdb34325bdcaed8a4d16ef67
I20260619 16:41:43.789857 8446 ts_tablet_manager.cc:1939] T 92e2914ebdb34325bdcaed8a4d16ef67 P 61292455bfa049e38d0cbda9e3d49542: Deleting consensus metadata
I20260619 16:41:43.790928 7895 catalog_manager.cc:4841] TS 61292455bfa049e38d0cbda9e3d49542 (127.0.98.3:44575): tablet 92e2914ebdb34325bdcaed8a4d16ef67 (table test-table [id=30bec7cfb3d34cbf8d35215558089e64]) successfully deleted
I20260619 16:41:43.792665 8424 tablet_bootstrap.cc:492] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542: No bootstrap required, opened a new log
I20260619 16:41:43.792809 8424 ts_tablet_manager.cc:1397] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20260619 16:41:43.793423 8424 raft_consensus.cc:348] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.793596 8424 raft_consensus.cc:374] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:43.793633 8424 raft_consensus.cc:729] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61292455bfa049e38d0cbda9e3d49542, State: Initialized, Role: FOLLOWER
I20260619 16:41:43.793845 8424 consensus_queue.cc:260] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.794312 8424 ts_tablet_manager.cc:1428] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260619 16:41:43.883561 8476 raft_consensus.cc:482] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:41:43.883816 8476 raft_consensus.cc:504] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.884362 8476 leader_election.cc:288] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a26293ac1ff74ce5a54b683d5f47954d (127.0.98.1:41255), b170ecc3958549ba8bf604b1b301c9f5 (127.0.98.2:43469)
I20260619 16:41:43.884938 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf985a6b6d2c43c5ad29126e47ab5c97" candidate_uuid: "61292455bfa049e38d0cbda9e3d49542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b170ecc3958549ba8bf604b1b301c9f5" is_pre_election: true
I20260619 16:41:43.884994 8068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf985a6b6d2c43c5ad29126e47ab5c97" candidate_uuid: "61292455bfa049e38d0cbda9e3d49542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a26293ac1ff74ce5a54b683d5f47954d" is_pre_election: true
I20260619 16:41:43.885120 8068 raft_consensus.cc:2454] T cf985a6b6d2c43c5ad29126e47ab5c97 P a26293ac1ff74ce5a54b683d5f47954d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61292455bfa049e38d0cbda9e3d49542 in term 0.
I20260619 16:41:43.885129 8208 raft_consensus.cc:2454] T cf985a6b6d2c43c5ad29126e47ab5c97 P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61292455bfa049e38d0cbda9e3d49542 in term 0.
I20260619 16:41:43.885459 8281 leader_election.cc:302] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61292455bfa049e38d0cbda9e3d49542, a26293ac1ff74ce5a54b683d5f47954d; no voters:
I20260619 16:41:43.885710 8476 raft_consensus.cc:2781] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:41:43.885830 8476 raft_consensus.cc:482] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:41:43.885871 8476 raft_consensus.cc:3037] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:43.887482 8476 raft_consensus.cc:504] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.888208 8476 leader_election.cc:288] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542 [CANDIDATE]: Term 1 election: Requested vote from peers a26293ac1ff74ce5a54b683d5f47954d (127.0.98.1:41255), b170ecc3958549ba8bf604b1b301c9f5 (127.0.98.2:43469)
I20260619 16:41:43.888525 8068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf985a6b6d2c43c5ad29126e47ab5c97" candidate_uuid: "61292455bfa049e38d0cbda9e3d49542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a26293ac1ff74ce5a54b683d5f47954d"
I20260619 16:41:43.888713 8068 raft_consensus.cc:3037] T cf985a6b6d2c43c5ad29126e47ab5c97 P a26293ac1ff74ce5a54b683d5f47954d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:43.888934 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf985a6b6d2c43c5ad29126e47ab5c97" candidate_uuid: "61292455bfa049e38d0cbda9e3d49542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b170ecc3958549ba8bf604b1b301c9f5"
I20260619 16:41:43.889103 8208 raft_consensus.cc:3037] T cf985a6b6d2c43c5ad29126e47ab5c97 P b170ecc3958549ba8bf604b1b301c9f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:43.889830 8068 raft_consensus.cc:2454] T cf985a6b6d2c43c5ad29126e47ab5c97 P a26293ac1ff74ce5a54b683d5f47954d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61292455bfa049e38d0cbda9e3d49542 in term 1.
I20260619 16:41:43.890211 8281 leader_election.cc:302] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61292455bfa049e38d0cbda9e3d49542, a26293ac1ff74ce5a54b683d5f47954d; no voters:
I20260619 16:41:43.890508 8208 raft_consensus.cc:2454] T cf985a6b6d2c43c5ad29126e47ab5c97 P b170ecc3958549ba8bf604b1b301c9f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61292455bfa049e38d0cbda9e3d49542 in term 1.
I20260619 16:41:43.890759 8476 raft_consensus.cc:2781] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:43.890995 8476 raft_consensus.cc:686] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542 [term 1 LEADER]: Becoming Leader. State: Replica: 61292455bfa049e38d0cbda9e3d49542, State: Running, Role: LEADER
I20260619 16:41:43.891196 8476 consensus_queue.cc:237] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } }
I20260619 16:41:43.894016 7909 catalog_manager.cc:5495] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61292455bfa049e38d0cbda9e3d49542 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "61292455bfa049e38d0cbda9e3d49542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61292455bfa049e38d0cbda9e3d49542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44575 } health_report { overall_health: HEALTHY } } }
I20260619 16:41:43.926463 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:43.922657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44206 (local address 127.0.98.3:44575)
0619 16:41:43.922813 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:43.922817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:43.922831 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:41:43.922898 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:43.922901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:43.922952 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:43.923033 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:41:43.923607 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.924303 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.925208 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:43.925513 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:43.925622 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:43.925911 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:41:43.926003 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:41:43.926072 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:43.926260 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 16:41:43.926315 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:41:43.935060 8068 raft_consensus.cc:1261] T cf985a6b6d2c43c5ad29126e47ab5c97 P a26293ac1ff74ce5a54b683d5f47954d [term 1 FOLLOWER]: Refusing update from remote peer 61292455bfa049e38d0cbda9e3d49542: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:43.935061 8208 raft_consensus.cc:1261] T cf985a6b6d2c43c5ad29126e47ab5c97 P b170ecc3958549ba8bf604b1b301c9f5 [term 1 FOLLOWER]: Refusing update from remote peer 61292455bfa049e38d0cbda9e3d49542: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:41:43.935506 8476 consensus_queue.cc:1035] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b170ecc3958549ba8bf604b1b301c9f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:43.935783 8428 consensus_queue.cc:1035] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a26293ac1ff74ce5a54b683d5f47954d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:41:43.974483 7909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:41:43.974735 7909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:41:43.977376 7909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81e280c69e54404f838b915627e155a8: Sending DeleteTablet for 3 replicas of tablet cf985a6b6d2c43c5ad29126e47ab5c97
I20260619 16:41:43.978411 8328 tablet_service.cc:1514] Processing DeleteTablet for tablet cf985a6b6d2c43c5ad29126e47ab5c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44182
I20260619 16:41:43.978392 8048 tablet_service.cc:1514] Processing DeleteTablet for tablet cf985a6b6d2c43c5ad29126e47ab5c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50444
I20260619 16:41:43.978392 8188 tablet_service.cc:1514] Processing DeleteTablet for tablet cf985a6b6d2c43c5ad29126e47ab5c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:41:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37470
I20260619 16:41:43.978667 8447 tablet_replica.cc:330] T cf985a6b6d2c43c5ad29126e47ab5c97 P a26293ac1ff74ce5a54b683d5f47954d: stopping tablet replica
I20260619 16:41:43.978732 8446 tablet_replica.cc:330] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542: stopping tablet replica
I20260619 16:41:43.978861 8447 raft_consensus.cc:2229] T cf985a6b6d2c43c5ad29126e47ab5c97 P a26293ac1ff74ce5a54b683d5f47954d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:41:43.978919 8446 raft_consensus.cc:2229] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:41:43.979074 8447 raft_consensus.cc:2258] T cf985a6b6d2c43c5ad29126e47ab5c97 P a26293ac1ff74ce5a54b683d5f47954d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:43.979279 8446 raft_consensus.cc:2258] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:43.979542 8448 tablet_replica.cc:330] T cf985a6b6d2c43c5ad29126e47ab5c97 P b170ecc3958549ba8bf604b1b301c9f5: stopping tablet replica
I20260619 16:41:43.979941 8448 raft_consensus.cc:2229] T cf985a6b6d2c43c5ad29126e47ab5c97 P b170ecc3958549ba8bf604b1b301c9f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:41:43.980079 8448 raft_consensus.cc:2258] T cf985a6b6d2c43c5ad29126e47ab5c97 P b170ecc3958549ba8bf604b1b301c9f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:41:43.980261 8447 ts_tablet_manager.cc:1905] T cf985a6b6d2c43c5ad29126e47ab5c97 P a26293ac1ff74ce5a54b683d5f47954d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:43.980499 8446 ts_tablet_manager.cc:1905] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:43.981101 8448 ts_tablet_manager.cc:1905] T cf985a6b6d2c43c5ad29126e47ab5c97 P b170ecc3958549ba8bf604b1b301c9f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:41:43.982898 8446 ts_tablet_manager.cc:1918] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:41:43.982998 8446 log.cc:1196] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-2/wal/wals/cf985a6b6d2c43c5ad29126e47ab5c97
I20260619 16:41:43.983289 8446 ts_tablet_manager.cc:1939] T cf985a6b6d2c43c5ad29126e47ab5c97 P 61292455bfa049e38d0cbda9e3d49542: Deleting consensus metadata
I20260619 16:41:43.983608 8447 ts_tablet_manager.cc:1918] T cf985a6b6d2c43c5ad29126e47ab5c97 P a26293ac1ff74ce5a54b683d5f47954d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:41:43.983733 8447 log.cc:1196] T cf985a6b6d2c43c5ad29126e47ab5c97 P a26293ac1ff74ce5a54b683d5f47954d: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-0/wal/wals/cf985a6b6d2c43c5ad29126e47ab5c97
I20260619 16:41:43.984122 8447 ts_tablet_manager.cc:1939] T cf985a6b6d2c43c5ad29126e47ab5c97 P a26293ac1ff74ce5a54b683d5f47954d: Deleting consensus metadata
I20260619 16:41:43.984087 7895 catalog_manager.cc:4841] TS 61292455bfa049e38d0cbda9e3d49542 (127.0.98.3:44575): tablet cf985a6b6d2c43c5ad29126e47ab5c97 (table test-table [id=82753187307d4e428461679b44d0582a]) successfully deleted
I20260619 16:41:43.984333 8448 ts_tablet_manager.cc:1918] T cf985a6b6d2c43c5ad29126e47ab5c97 P b170ecc3958549ba8bf604b1b301c9f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:41:43.984421 8448 log.cc:1196] T cf985a6b6d2c43c5ad29126e47ab5c97 P b170ecc3958549ba8bf604b1b301c9f5: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887259574980-392-0/minicluster-data/ts-1/wal/wals/cf985a6b6d2c43c5ad29126e47ab5c97
I20260619 16:41:43.984680 8448 ts_tablet_manager.cc:1939] T cf985a6b6d2c43c5ad29126e47ab5c97 P b170ecc3958549ba8bf604b1b301c9f5: Deleting consensus metadata
I20260619 16:41:43.985096 7897 catalog_manager.cc:4841] TS a26293ac1ff74ce5a54b683d5f47954d (127.0.98.1:41255): tablet cf985a6b6d2c43c5ad29126e47ab5c97 (table test-table [id=82753187307d4e428461679b44d0582a]) successfully deleted
I20260619 16:41:43.985139 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 7983
I20260619 16:41:43.986603 7895 catalog_manager.cc:4841] TS b170ecc3958549ba8bf604b1b301c9f5 (127.0.98.2:43469): tablet cf985a6b6d2c43c5ad29126e47ab5c97 (table test-table [id=82753187307d4e428461679b44d0582a]) successfully deleted
I20260619 16:41:43.992713 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 8123
I20260619 16:41:44.001361 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 8263
I20260619 16:41:44.010830 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 7879
2026-06-19T16:41:44Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3578 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:41:44 dist-test-slave-03cd krb5kdc[8508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:41:44 dist-test-slave-03cd krb5kdc[8508](info): set up 2 sockets
Jun 19 16:41:44 dist-test-slave-03cd krb5kdc[8508](info): commencing operation
krb5kdc: starting...
W20260619 16:41:45.116958 392 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:41:45 dist-test-slave-03cd krb5kdc[8508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:41:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:41:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:41:45.265136 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40747
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:45.408192 8521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:45.408684 8521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:45.408803 8521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:45.413767 8521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:41:45.413889 8521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:45.413911 8521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:45.413930 8521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:41:45.413949 8521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:41:45.420823 8521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:45.422480 8521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:45.423918 8521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:45.435494 8529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:45.435587 8521 server_base.cc:1034] running on GCE node
W20260619 16:41:45.435559 8527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:45.435508 8526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:45.436759 8521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:45.438277 8521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:45.439491 8521 hybrid_clock.cc:648] HybridClock initialized: now 1781887305439452 us; error 65 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T16:41:45Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1433 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:41:45 dist-test-slave-03cd krb5kdc[8534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:41:45 dist-test-slave-03cd krb5kdc[8534](info): set up 2 sockets
Jun 19 16:41:45 dist-test-slave-03cd krb5kdc[8534](info): commencing operation
krb5kdc: starting...
W20260619 16:41:46.517114 392 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:41:46 dist-test-slave-03cd krb5kdc[8534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:41:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:41:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:41:46.664273 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39789
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:46.803781 8547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:46.804437 8547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:46.804592 8547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:46.810467 8547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:41:46.810669 8547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:46.810716 8547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:46.810755 8547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:41:46.810797 8547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:41:46.817873 8547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:46.819475 8547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:46.820888 8547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:46.831753 8555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:46.831822 8547 server_base.cc:1034] running on GCE node
W20260619 16:41:46.831753 8553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:46.831789 8552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:46.832870 8547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:46.834232 8547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:46.835457 8547 hybrid_clock.cc:648] HybridClock initialized: now 1781887306835440 us; error 45 us; skew 500 ppm
Jun 19 16:41:46 dist-test-slave-03cd krb5kdc[8534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:46.838727 8547 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:41:46.840392 8547 webserver.cc:460] Webserver started at http://127.0.98.62:36771/ using document root <none> and password file <none>
I20260619 16:41:46.841279 8547 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:46.841384 8547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:46.841635 8547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:46.844134 8547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "0fd94caa60e54ea6a49c96298ffe80c8"
format_stamp: "Formatted at 2026-06-19 16:41:46 on dist-test-slave-03cd"
server_key: "aadfacaeb9cc0635ab2a0750087688ce"
server_key_iv: "5da840ee9dfffd26c264766eb5cffeb1"
server_key_version: "encryptionkey@0"
I20260619 16:41:46.844805 8547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "0fd94caa60e54ea6a49c96298ffe80c8"
format_stamp: "Formatted at 2026-06-19 16:41:46 on dist-test-slave-03cd"
server_key: "aadfacaeb9cc0635ab2a0750087688ce"
server_key_iv: "5da840ee9dfffd26c264766eb5cffeb1"
server_key_version: "encryptionkey@0"
I20260619 16:41:46.849624 8547 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:41:46.853015 8562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:46.855485 8547 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:41:46.855779 8547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "0fd94caa60e54ea6a49c96298ffe80c8"
format_stamp: "Formatted at 2026-06-19 16:41:46 on dist-test-slave-03cd"
server_key: "aadfacaeb9cc0635ab2a0750087688ce"
server_key_iv: "5da840ee9dfffd26c264766eb5cffeb1"
server_key_version: "encryptionkey@0"
I20260619 16:41:46.856050 8547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:46.876955 8547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:46.881397 8547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:46.881685 8547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:46.892379 8547 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39789
I20260619 16:41:46.892403 8614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39789 every 8 connection(s)
I20260619 16:41:46.893934 8547 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:41:46.898525 8615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:41:46.904163 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 8547
I20260619 16:41:46.904316 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:41:46.904677 392 external_mini_cluster.cc:1428] Setting key 80f5868493e62c1f81002d7a225ca2e4
I20260619 16:41:46.905378 8615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8: Bootstrap starting.
I20260619 16:41:46.908404 8615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8: Neither blocks nor log segments found. Creating new log.
I20260619 16:41:46.909576 8615 log.cc:824] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:46.912886 8615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8: No bootstrap required, opened a new log
Jun 19 16:41:46 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:46.918602 8615 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd94caa60e54ea6a49c96298ffe80c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39789 } }
I20260619 16:41:46.919004 8615 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:41:46.919063 8615 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fd94caa60e54ea6a49c96298ffe80c8, State: Initialized, Role: FOLLOWER
I20260619 16:41:46.919775 8615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd94caa60e54ea6a49c96298ffe80c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39789 } }
I20260619 16:41:46.920002 8615 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:41:46.920094 8615 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:41:46.920233 8615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:41:46.921798 8615 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd94caa60e54ea6a49c96298ffe80c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39789 } }
I20260619 16:41:46.922302 8615 leader_election.cc:302] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fd94caa60e54ea6a49c96298ffe80c8; no voters:
I20260619 16:41:46.922771 8615 leader_election.cc:288] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:41:46.922927 8620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:41:46.923169 8620 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [term 1 LEADER]: Becoming Leader. State: Replica: 0fd94caa60e54ea6a49c96298ffe80c8, State: Running, Role: LEADER
I20260619 16:41:46.923552 8620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd94caa60e54ea6a49c96298ffe80c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39789 } }
I20260619 16:41:46.924530 8615 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:41:46.925696 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:46.906834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38326 (local address 127.0.98.62:39789)
0619 16:41:46.907513 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:46.907548 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:46.907613 (+ 65us) server_negotiation.cc:407] Connection header received
0619 16:41:46.909165 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:46.909196 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:46.909962 (+ 766us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:46.910436 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:46.911389 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:46.913051 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:46.914403 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:46.915253 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:46.915475 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:46.918938 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:46.918947 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:46.918974 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:46.919031 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:46.921563 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:46.922269 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:46.922278 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:46.922285 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:46.922532 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:46.922928 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:46.922931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:46.922933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:46.923707 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:46.923939 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:46.924241 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 16:41:46.924599 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":215,"threads_started":1}
I20260619 16:41:46.926075 8622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fd94caa60e54ea6a49c96298ffe80c8. Latest consensus state: current_term: 1 leader_uuid: "0fd94caa60e54ea6a49c96298ffe80c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd94caa60e54ea6a49c96298ffe80c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39789 } } }
I20260619 16:41:46.926165 8621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fd94caa60e54ea6a49c96298ffe80c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd94caa60e54ea6a49c96298ffe80c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39789 } } }
I20260619 16:41:46.926307 8621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:46.926241 8622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:46.927359 8629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:41:46.931080 8629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:41:46.939065 8629 catalog_manager.cc:1347] Generated new cluster ID: 236fe4c7e390491e88ae3e35424c5926
I20260619 16:41:46.939193 8629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:41:46.950811 8629 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:41:46.952147 8629 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:41:46.963384 8629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8: Generated new TSK 0
I20260619 16:41:46.964447 8629 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:41:47.012604 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39789
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:47.158738 8641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:47.159118 8641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:47.159170 8641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:47.164149 8641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:47.164256 8641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:47.164330 8641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:41:47.170495 8641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:47.172093 8641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:47.173471 8641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:47.187575 8646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:47.188012 8649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:47.188015 8647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:47.188223 8641 server_base.cc:1034] running on GCE node
I20260619 16:41:47.188836 8641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:47.189814 8641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:47.191062 8641 hybrid_clock.cc:648] HybridClock initialized: now 1781887307191013 us; error 63 us; skew 500 ppm
Jun 19 16:41:47 dist-test-slave-03cd krb5kdc[8534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:47.194532 8641 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:41:47.195779 8641 webserver.cc:460] Webserver started at http://127.0.98.1:42813/ using document root <none> and password file <none>
I20260619 16:41:47.196609 8641 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:47.196724 8641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:47.196996 8641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:47.199440 8641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/data/instance:
uuid: "7dea2300cdc4467da20237165b7b00f3"
format_stamp: "Formatted at 2026-06-19 16:41:47 on dist-test-slave-03cd"
server_key: "e9ce3b2cc562eb58eca6b73df92d7d55"
server_key_iv: "bea0a1cd8e2a73a27803e12adcf41ed5"
server_key_version: "encryptionkey@0"
I20260619 16:41:47.200176 8641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7dea2300cdc4467da20237165b7b00f3"
format_stamp: "Formatted at 2026-06-19 16:41:47 on dist-test-slave-03cd"
server_key: "e9ce3b2cc562eb58eca6b73df92d7d55"
server_key_iv: "bea0a1cd8e2a73a27803e12adcf41ed5"
server_key_version: "encryptionkey@0"
I20260619 16:41:47.204841 8641 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:41:47.207851 8656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:47.209584 8641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:41:47.209810 8641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "7dea2300cdc4467da20237165b7b00f3"
format_stamp: "Formatted at 2026-06-19 16:41:47 on dist-test-slave-03cd"
server_key: "e9ce3b2cc562eb58eca6b73df92d7d55"
server_key_iv: "bea0a1cd8e2a73a27803e12adcf41ed5"
server_key_version: "encryptionkey@0"
I20260619 16:41:47.209971 8641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:47.260319 8641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:47.265919 8641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:47.266163 8641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:47.266961 8641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:47.268177 8641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:47.268242 8641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:47.268323 8641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:47.268352 8641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:47.278685 8641 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45613
I20260619 16:41:47.278705 8769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45613 every 8 connection(s)
I20260619 16:41:47.279842 8641 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:41:47.282874 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 8641
I20260619 16:41:47.283020 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:41:47.283511 392 external_mini_cluster.cc:1428] Setting key c3e41106ef48c172c68c9d17d307577f
Jun 19 16:41:47 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 16:41:47.298439 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:47.281756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40623 (local address 127.0.98.62:39789)
0619 16:41:47.281925 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:47.281928 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:47.282970 (+ 1042us) server_negotiation.cc:407] Connection header received
0619 16:41:47.284858 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:47.284862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:47.284913 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:47.285025 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:47.287167 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:47.288121 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:47.289560 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:47.289984 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:47.290121 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:47.293416 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:47.293423 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:47.293427 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:47.293461 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:47.295142 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:47.296148 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:47.296151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:47.296152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:47.296247 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:47.296869 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:47.296872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:47.296874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:47.297117 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:47.297258 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:47.298070 (+ 812us) server_negotiation.cc:299] Negotiation successful
0619 16:41:47.298226 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 16:41:47.299363 8772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:47.282087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.1:40623)
0619 16:41:47.282642 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:47.282705 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:47.284608 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:47.285291 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:47.285299 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:47.286006 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:47.286892 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:47.286918 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:47.288273 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:47.288276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:47.289414 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:47.289428 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:47.290245 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:47.290252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:47.290473 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:47.291268 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:47.291288 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:47.293170 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:47.295288 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:47.295302 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:47.295318 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:47.295779 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:47.296404 (+ 625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:47.296407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:47.296408 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:47.296709 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:47.297296 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:47.297336 (+ 40us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:47.297764 (+ 428us) client_negotiation.cc:769] Sending connection context
0619 16:41:47.298013 (+ 249us) client_negotiation.cc:240] Negotiation successful
0619 16:41:47.298395 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":118,"threads_started":1}
I20260619 16:41:47.301156 8770 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39789
I20260619 16:41:47.301468 8770 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:47.302095 8770 heartbeater.cc:511] Master 127.0.98.62:39789 requested a full tablet report, sending...
I20260619 16:41:47.304096 8579 ts_manager.cc:194] Registered new tserver with Master: 7dea2300cdc4467da20237165b7b00f3 (127.0.98.1:45613)
I20260619 16:41:47.305843 8579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:41:47.320439 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39789
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:47.444855 8775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:47.445200 8775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:47.445277 8775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:47.449941 8775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:47.450109 8775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:47.450215 8775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:41:47.456291 8775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:47.457594 8775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:47.458973 8775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:47.470655 8781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:47.470717 8775 server_base.cc:1034] running on GCE node
W20260619 16:41:47.470669 8780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:47.470669 8783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:47.471464 8775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:47.472256 8775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:47.473433 8775 hybrid_clock.cc:648] HybridClock initialized: now 1781887307473418 us; error 28 us; skew 500 ppm
Jun 19 16:41:47 dist-test-slave-03cd krb5kdc[8534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:47.476720 8775 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:41:47.477695 8775 webserver.cc:460] Webserver started at http://127.0.98.2:37665/ using document root <none> and password file <none>
I20260619 16:41:47.478315 8775 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:47.478382 8775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:47.478602 8775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:47.480856 8775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/data/instance:
uuid: "2f26a7a66f174567a38136de967197fd"
format_stamp: "Formatted at 2026-06-19 16:41:47 on dist-test-slave-03cd"
server_key: "2e4b4bb850db65aa11f5078d0516a534"
server_key_iv: "1b6adc6b9da2dbabd04d7575df6bfec4"
server_key_version: "encryptionkey@0"
I20260619 16:41:47.481478 8775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2f26a7a66f174567a38136de967197fd"
format_stamp: "Formatted at 2026-06-19 16:41:47 on dist-test-slave-03cd"
server_key: "2e4b4bb850db65aa11f5078d0516a534"
server_key_iv: "1b6adc6b9da2dbabd04d7575df6bfec4"
server_key_version: "encryptionkey@0"
I20260619 16:41:47.485652 8775 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:41:47.488556 8790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:47.490082 8775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:41:47.490218 8775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "2f26a7a66f174567a38136de967197fd"
format_stamp: "Formatted at 2026-06-19 16:41:47 on dist-test-slave-03cd"
server_key: "2e4b4bb850db65aa11f5078d0516a534"
server_key_iv: "1b6adc6b9da2dbabd04d7575df6bfec4"
server_key_version: "encryptionkey@0"
I20260619 16:41:47.490348 8775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:47.510915 8775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:47.515010 8775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:47.515264 8775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:47.515983 8775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:47.516855 8775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:47.516903 8775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:47.516970 8775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:47.517004 8775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:47.527276 8775 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44877
I20260619 16:41:47.527388 8903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44877 every 8 connection(s)
I20260619 16:41:47.528313 8775 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:41:47.537953 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 8775
I20260619 16:41:47.538058 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:41:47.538415 392 external_mini_cluster.cc:1428] Setting key 046161927af14f803bdf2da72f3c8f1e
Jun 19 16:41:47 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:47.543780 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:47.530409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46829 (local address 127.0.98.62:39789)
0619 16:41:47.530579 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:47.530582 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:47.531548 (+ 966us) server_negotiation.cc:407] Connection header received
0619 16:41:47.533122 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:47.533126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:47.533188 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:47.533301 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:47.534567 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:47.535116 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:47.536321 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:47.536560 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:47.536680 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:47.539641 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:47.539645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:47.539648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:47.539697 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:47.541383 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:47.542099 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:47.542104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:47.542105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:47.542194 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:47.542575 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:47.542580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:47.542581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:47.542784 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:47.542897 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:47.543439 (+ 542us) server_negotiation.cc:299] Negotiation successful
0619 16:41:47.543572 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 16:41:47.544754 8906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:47.530883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.2:46829)
0619 16:41:47.531379 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:47.531426 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:47.532881 (+ 1455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:47.533491 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:47.533498 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:47.533897 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:47.534405 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:47.534420 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:47.535251 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:47.535257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:47.536188 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:47.536198 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:47.536730 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:47.536734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:47.536925 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:47.537574 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:47.537591 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:47.539462 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:47.541519 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:47.541529 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:47.541552 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:47.541991 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:47.542293 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:47.542298 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:47.542300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:47.542467 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:47.542899 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:47.542908 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:47.543190 (+ 282us) client_negotiation.cc:769] Sending connection context
0619 16:41:47.543400 (+ 210us) client_negotiation.cc:240] Negotiation successful
0619 16:41:47.543806 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":121,"threads_started":1}
I20260619 16:41:47.546087 8904 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39789
I20260619 16:41:47.546389 8904 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:47.547107 8904 heartbeater.cc:511] Master 127.0.98.62:39789 requested a full tablet report, sending...
I20260619 16:41:47.548527 8579 ts_manager.cc:194] Registered new tserver with Master: 2f26a7a66f174567a38136de967197fd (127.0.98.2:44877)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:41:47.549258 8579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:41:47.572335 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39789
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:47.694747 8909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:47.695138 8909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:47.695189 8909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:47.699878 8909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:47.699959 8909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:47.700024 8909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:41:47.705651 8909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:47.706743 8909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:47.708328 8909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:41:47.719388 8909 server_base.cc:1034] running on GCE node
W20260619 16:41:47.719333 8914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:47.719333 8915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:47.719326 8917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:47.720144 8909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:47.720782 8909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:47.721999 8909 hybrid_clock.cc:648] HybridClock initialized: now 1781887307721949 us; error 49 us; skew 500 ppm
Jun 19 16:41:47 dist-test-slave-03cd krb5kdc[8534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:47.726167 8909 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:41:47.727320 8909 webserver.cc:460] Webserver started at http://127.0.98.3:43759/ using document root <none> and password file <none>
I20260619 16:41:47.728029 8909 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:47.728097 8909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:47.728325 8909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:41:47.730495 8909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/data/instance:
uuid: "ccc0bbdda6604412835fd41efaa960e9"
format_stamp: "Formatted at 2026-06-19 16:41:47 on dist-test-slave-03cd"
server_key: "edbfda782f480b2b849ddd4884bec22b"
server_key_iv: "94102e1681d186fe740f21f1d4a855ba"
server_key_version: "encryptionkey@0"
I20260619 16:41:47.731102 8909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ccc0bbdda6604412835fd41efaa960e9"
format_stamp: "Formatted at 2026-06-19 16:41:47 on dist-test-slave-03cd"
server_key: "edbfda782f480b2b849ddd4884bec22b"
server_key_iv: "94102e1681d186fe740f21f1d4a855ba"
server_key_version: "encryptionkey@0"
I20260619 16:41:47.735174 8909 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260619 16:41:47.737802 8924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:47.739208 8909 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:41:47.739318 8909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "ccc0bbdda6604412835fd41efaa960e9"
format_stamp: "Formatted at 2026-06-19 16:41:47 on dist-test-slave-03cd"
server_key: "edbfda782f480b2b849ddd4884bec22b"
server_key_iv: "94102e1681d186fe740f21f1d4a855ba"
server_key_version: "encryptionkey@0"
I20260619 16:41:47.739425 8909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:47.750766 8909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:47.754269 8909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:47.754453 8909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:47.755019 8909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:47.755892 8909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:47.755949 8909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:47.755990 8909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:47.756006 8909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:47.765537 8909 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36011
I20260619 16:41:47.765682 9037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36011 every 8 connection(s)
I20260619 16:41:47.766499 8909 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:41:47.769521 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 8909
I20260619 16:41:47.769666 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:41:47.769991 392 external_mini_cluster.cc:1428] Setting key c795f05205622101aeb7f762ae94e801
Jun 19 16:41:47 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:41:47.782003 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:47.768544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35047 (local address 127.0.98.62:39789)
0619 16:41:47.768730 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:47.768735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:47.769598 (+ 863us) server_negotiation.cc:407] Connection header received
0619 16:41:47.771532 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:47.771536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:47.771597 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:47.771709 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:47.773131 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:47.773682 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:47.774902 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:47.775126 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:47.775248 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:47.778160 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:47.778163 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:47.778166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:47.778192 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:47.779830 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:47.780403 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:47.780407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:47.780409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:47.780498 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:47.780843 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:47.780848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:47.780850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:47.781060 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:47.781167 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:47.781698 (+ 531us) server_negotiation.cc:299] Negotiation successful
0619 16:41:47.781831 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 16:41:47.782867 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:47.768939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.3:35047)
0619 16:41:47.769429 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:47.769476 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:47.771280 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:47.771916 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:47.771925 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:47.772396 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:47.772973 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:47.772987 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:47.773819 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:47.773822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:47.774748 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:47.774758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:47.775296 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:47.775300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:47.775489 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:47.776249 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:47.776266 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:47.777908 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:41:47.779970 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:47.779977 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:47.779997 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:47.780288 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:47.780592 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:41:47.780596 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:41:47.780598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:41:47.780747 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:41:47.781177 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:41:47.781186 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:41:47.781457 (+ 271us) client_negotiation.cc:769] Sending connection context
0619 16:41:47.781660 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 16:41:47.782015 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":127,"threads_started":1}
I20260619 16:41:47.783959 9038 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39789
I20260619 16:41:47.784215 9038 heartbeater.cc:464] Registering TS with master...
I20260619 16:41:47.784771 9038 heartbeater.cc:511] Master 127.0.98.62:39789 requested a full tablet report, sending...
I20260619 16:41:47.785821 8579 ts_manager.cc:194] Registered new tserver with Master: ccc0bbdda6604412835fd41efaa960e9 (127.0.98.3:36011)
I20260619 16:41:47.786438 8579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35047
I20260619 16:41:47.796860 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 16:41:47.883821 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 8641
I20260619 16:41:47.889626 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 8775
I20260619 16:41:47.895008 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 8909
I20260619 16:41:47.901636 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 8547
I20260619 16:41:47.908113 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39789
--webserver_interface=127.0.98.62
--webserver_port=36771
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:41:48.029155 9047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:48.029536 9047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:48.029624 9047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:48.033907 9047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:41:48.034034 9047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:48.034072 9047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:48.034206 9047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:41:48.034289 9047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:41:48.040396 9047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:48.041708 9047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:48.042824 9047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:41:48.052440 9047 server_base.cc:1034] running on GCE node
W20260619 16:41:48.052399 9052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:48.052399 9053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:48.052399 9055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:48.053357 9047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:48.054621 9047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:48.055786 9047 hybrid_clock.cc:648] HybridClock initialized: now 1781887308055755 us; error 43 us; skew 500 ppm
Jun 19 16:41:48 dist-test-slave-03cd krb5kdc[8534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887308, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:48.058859 9047 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:41:48.059885 9047 webserver.cc:460] Webserver started at http://127.0.98.62:36771/ using document root <none> and password file <none>
I20260619 16:41:48.060434 9047 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:48.060509 9047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:48.064514 9047 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:41:48.066753 9062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:48.068148 9047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:41:48.068279 9047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "0fd94caa60e54ea6a49c96298ffe80c8"
format_stamp: "Formatted at 2026-06-19 16:41:46 on dist-test-slave-03cd"
server_key: "aadfacaeb9cc0635ab2a0750087688ce"
server_key_iv: "5da840ee9dfffd26c264766eb5cffeb1"
server_key_version: "encryptionkey@0"
I20260619 16:41:48.068850 9047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:48.090181 9047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:48.093961 9047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:48.094237 9047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:48.102696 9047 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39789
I20260619 16:41:48.102711 9114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39789 every 8 connection(s)
I20260619 16:41:48.103965 9047 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:41:48.105796 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 9047
I20260619 16:41:48.106277 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45613
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39789
--webserver_port=42813
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:41:48.109201 9115 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:41:48.112107 9115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8: Bootstrap starting.
I20260619 16:41:48.136236 9115 log.cc:824] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8: Log is configured to *not* fsync() on all Append() calls
I20260619 16:41:48.144961 9115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:41:48.145421 9115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8: Bootstrap complete.
I20260619 16:41:48.151252 9115 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd94caa60e54ea6a49c96298ffe80c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39789 } }
I20260619 16:41:48.151859 9115 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fd94caa60e54ea6a49c96298ffe80c8, State: Initialized, Role: FOLLOWER
I20260619 16:41:48.152487 9115 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd94caa60e54ea6a49c96298ffe80c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39789 } }
I20260619 16:41:48.152670 9115 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:41:48.152755 9115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:41:48.152858 9115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:41:48.155565 9115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd94caa60e54ea6a49c96298ffe80c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39789 } }
I20260619 16:41:48.156025 9115 leader_election.cc:302] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fd94caa60e54ea6a49c96298ffe80c8; no voters:
I20260619 16:41:48.156433 9115 leader_election.cc:288] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 16:41:48.156587 9119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:41:48.156875 9119 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [term 2 LEADER]: Becoming Leader. State: Replica: 0fd94caa60e54ea6a49c96298ffe80c8, State: Running, Role: LEADER
I20260619 16:41:48.157202 9119 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd94caa60e54ea6a49c96298ffe80c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39789 } }
I20260619 16:41:48.157720 9115 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:41:48.158815 9121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fd94caa60e54ea6a49c96298ffe80c8. Latest consensus state: current_term: 2 leader_uuid: "0fd94caa60e54ea6a49c96298ffe80c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd94caa60e54ea6a49c96298ffe80c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39789 } } }
I20260619 16:41:48.158812 9120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0fd94caa60e54ea6a49c96298ffe80c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd94caa60e54ea6a49c96298ffe80c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39789 } } }
I20260619 16:41:48.158972 9120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:48.158972 9121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:41:48.159555 9128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:41:48.163187 9128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:41:48.163957 9128 catalog_manager.cc:1259] Loaded cluster ID: 236fe4c7e390491e88ae3e35424c5926
I20260619 16:41:48.164022 9128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:41:48.166426 9128 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:41:48.167117 9128 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0fd94caa60e54ea6a49c96298ffe80c8: Loaded TSK: 0
I20260619 16:41:48.168414 9128 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:41:48.243109 9117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:48.243409 9117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:48.243455 9117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:48.247866 9117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:48.247954 9117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:48.248044 9117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:41:48.253652 9117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42813
--tserver_master_addrs=127.0.98.62:39789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:48.254808 9117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:48.256083 9117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:48.266896 9141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:48.267169 9117 server_base.cc:1034] running on GCE node
W20260619 16:41:48.266949 9142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:48.266947 9144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:48.268062 9117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:48.268908 9117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:48.270115 9117 hybrid_clock.cc:648] HybridClock initialized: now 1781887308270092 us; error 32 us; skew 500 ppm
Jun 19 16:41:48 dist-test-slave-03cd krb5kdc[8534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887308, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:48.273973 9117 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 16:41:48.275345 9117 webserver.cc:460] Webserver started at http://127.0.98.1:42813/ using document root <none> and password file <none>
I20260619 16:41:48.276131 9117 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:48.276266 9117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:48.281806 9117 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 16:41:48.284657 9151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:48.286262 9117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:41:48.286403 9117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "7dea2300cdc4467da20237165b7b00f3"
format_stamp: "Formatted at 2026-06-19 16:41:47 on dist-test-slave-03cd"
server_key: "e9ce3b2cc562eb58eca6b73df92d7d55"
server_key_iv: "bea0a1cd8e2a73a27803e12adcf41ed5"
server_key_version: "encryptionkey@0"
I20260619 16:41:48.287153 9117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:48.314996 9117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:48.319330 9117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:48.319576 9117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:48.320361 9117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:48.321378 9117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:48.321439 9117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:48.321491 9117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:48.321511 9117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:48.331616 9117 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45613
I20260619 16:41:48.331634 9264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45613 every 8 connection(s)
I20260619 16:41:48.332768 9117 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:41:48.335621 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 9117
I20260619 16:41:48.336246 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44877
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39789
--webserver_port=37665
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:41:48 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:48 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:48.349664 9268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:48.335170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.1:43467)
0619 16:41:48.335903 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:48.335967 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:48.337956 (+ 1989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:48.339498 (+ 1542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:48.339506 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:48.339989 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:48.340471 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:48.340491 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.342704 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.342707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:48.343805 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:48.343817 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.344653 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.344657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:48.344867 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:48.345531 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:48.345548 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:48.348799 (+ 3251us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":139,"threads_started":1}
W20260619 16:41:48.349946 9267 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:48.335092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43467 (local address 127.0.98.62:39789)
0619 16:41:48.335642 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:48.335695 (+ 53us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:48.336116 (+ 421us) server_negotiation.cc:407] Connection header received
0619 16:41:48.338563 (+ 2447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:48.338574 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:48.338990 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:48.339374 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:48.340683 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.342494 (+ 1811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.343957 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.344456 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.344656 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:48.349102 (+ 4446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43467: BlockingRecv error: recv got EOF from 127.0.98.1:43467 (error 108)
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":160,"threads_started":1}
W20260619 16:41:48.350498 9265 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39789 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:39789: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 16:41:48 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:48 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:48.357688 9268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:48.351032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.1:34493)
0619 16:41:48.351202 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:48.351218 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:48.351314 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:48.351571 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:48.351574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:48.351807 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:48.351932 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:48.351940 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.352725 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.352728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:48.353648 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:48.353659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.354164 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.354171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:48.354472 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:48.355077 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:48.355093 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:48.357548 (+ 2455us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":85}
W20260619 16:41:48.358119 9267 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:48.351159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34493 (local address 127.0.98.62:39789)
0619 16:41:48.351280 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:48.351284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:48.351299 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:41:48.351414 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:48.351416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:48.351460 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:48.351531 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:48.352065 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.352618 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.353787 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.354033 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.354121 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:48.357966 (+ 3845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34493: BlockingRecv error: recv got EOF from 127.0.98.1:34493 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 16:41:48 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:48 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:48.365414 9268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:48.358442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.1:42939)
0619 16:41:48.358618 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:48.358638 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:48.358724 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:48.359090 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:48.359093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:48.359237 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:48.359331 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:48.359337 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.360523 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.360526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:48.361350 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:48.361363 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.362032 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.362035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:48.362104 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:48.362701 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:48.362713 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:48.365284 (+ 2571us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 16:41:48.365671 9267 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:48.358680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42939 (local address 127.0.98.62:39789)
0619 16:41:48.358805 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:48.358811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:48.358827 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:41:48.358875 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:48.358878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:48.358928 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:48.359021 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:48.359451 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.360322 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.361505 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.361844 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.361965 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:48.365538 (+ 3573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42939: BlockingRecv error: recv got EOF from 127.0.98.1:42939 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260619 16:41:48.365711 9265 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:41:48.464478 9269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:48.464772 9269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:48.464824 9269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:48.469563 9269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:48.469661 9269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:48.469769 9269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:41:48.476440 9269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37665
--tserver_master_addrs=127.0.98.62:39789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:48.477669 9269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:48.478785 9269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:48.489365 9275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:48.489387 9274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:48.489434 9277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:48.489692 9269 server_base.cc:1034] running on GCE node
I20260619 16:41:48.490260 9269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:48.491021 9269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:48.492237 9269 hybrid_clock.cc:648] HybridClock initialized: now 1781887308492216 us; error 33 us; skew 500 ppm
Jun 19 16:41:48 dist-test-slave-03cd krb5kdc[8534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887308, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:48.499920 9269 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 16:41:48.500890 9269 webserver.cc:460] Webserver started at http://127.0.98.2:37665/ using document root <none> and password file <none>
I20260619 16:41:48.501461 9269 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:48.501536 9269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:48.505888 9269 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:41:48.508308 9284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:48.509733 9269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:41:48.509852 9269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "2f26a7a66f174567a38136de967197fd"
format_stamp: "Formatted at 2026-06-19 16:41:47 on dist-test-slave-03cd"
server_key: "2e4b4bb850db65aa11f5078d0516a534"
server_key_iv: "1b6adc6b9da2dbabd04d7575df6bfec4"
server_key_version: "encryptionkey@0"
I20260619 16:41:48.510473 9269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:48.525583 9269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:48.529127 9269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:48.529340 9269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:48.530159 9269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:48.531073 9269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:48.531122 9269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:48.531193 9269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:48.531229 9269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:48.540683 9269 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44877
I20260619 16:41:48.540699 9397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44877 every 8 connection(s)
I20260619 16:41:48.541759 9269 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:41:48.544873 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 9269
I20260619 16:41:48.545343 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36011
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39789
--webserver_port=43759
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:41:48 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:48 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 16:41:48.554320 9267 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:48.543934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40045 (local address 127.0.98.62:39789)
0619 16:41:48.544110 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:48.544114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:48.545083 (+ 969us) server_negotiation.cc:407] Connection header received
0619 16:41:48.546673 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:48.546677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:48.546746 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:48.546856 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:48.548203 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.548741 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.549963 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.550196 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.550320 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:48.554185 (+ 3865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40045: BlockingRecv error: recv got EOF from 127.0.98.2:40045 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:41:48.555173 9400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:48.544327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.2:40045)
0619 16:41:48.544893 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:48.544969 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:48.546427 (+ 1458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:48.547049 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:48.547056 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:48.547443 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:48.548031 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:48.548047 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.548886 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.548891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:48.549810 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:48.549822 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.550376 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.550381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:48.550654 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:48.551398 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:48.551415 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:48.554040 (+ 2625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":139,"threads_started":1}
W20260619 16:41:48.555828 9398 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39789 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:39789: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 16:41:48 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:48 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:48.562742 9400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:48.556377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.2:40953)
0619 16:41:48.556582 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:48.556597 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:48.556682 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:48.556940 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:48.556943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:48.557139 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:48.557275 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:48.557281 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.558128 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.558132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:48.559186 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:48.559199 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.559811 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.559815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:48.559919 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:48.560506 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:48.560521 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:48.562628 (+ 2107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133}
W20260619 16:41:48.563319 9267 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:48.556536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40953 (local address 127.0.98.62:39789)
0619 16:41:48.556661 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:48.556664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:48.556679 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:41:48.556776 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:48.556778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:48.556818 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:48.556895 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:48.557458 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.557988 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.559323 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.559582 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.559789 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:48.562852 (+ 3063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40953: BlockingRecv error: recv got EOF from 127.0.98.2:40953 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 16:41:48 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:48 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:48.569484 9400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:48.563546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.2:53811)
0619 16:41:48.563714 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:48.563729 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:48.563812 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:48.564256 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:48.564259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:48.564404 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:48.564490 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:48.564495 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.565705 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.565708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:48.566281 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:48.566290 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.566783 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.566787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:48.566884 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:48.567415 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:48.567428 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:48.569317 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 16:41:48.569545 9267 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:48.563862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53811 (local address 127.0.98.62:39789)
0619 16:41:48.563985 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:48.563989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:48.564003 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:41:48.564062 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:48.564065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:48.564108 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:48.564193 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:48.564717 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.565566 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.566445 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.566666 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.566780 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:48.569438 (+ 2658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53811: BlockingRecv error: recv got EOF from 127.0.98.2:53811 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260619 16:41:48.569761 9398 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:41:48.667407 9401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:41:48.667797 9401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:41:48.667886 9401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:41:48.672391 9401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:41:48.672457 9401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:41:48.672556 9401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:41:48.678409 9401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36011
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43759
--tserver_master_addrs=127.0.98.62:39789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:41:48.679548 9401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:41:48.680589 9401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:41:48.690482 9407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:48.690488 9406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:41:48.690570 9409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:41:48.690734 9401 server_base.cc:1034] running on GCE node
I20260619 16:41:48.691262 9401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:41:48.692116 9401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:41:48.693320 9401 hybrid_clock.cc:648] HybridClock initialized: now 1781887308693278 us; error 72 us; skew 500 ppm
Jun 19 16:41:48 dist-test-slave-03cd krb5kdc[8534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887308, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:41:48.696317 9401 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 16:41:48.697345 9401 webserver.cc:460] Webserver started at http://127.0.98.3:43759/ using document root <none> and password file <none>
I20260619 16:41:48.697925 9401 fs_manager.cc:359] Metadata directory not provided
I20260619 16:41:48.698046 9401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:41:48.702050 9401 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:41:48.704360 9416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:48.705849 9401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:41:48.706027 9401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "ccc0bbdda6604412835fd41efaa960e9"
format_stamp: "Formatted at 2026-06-19 16:41:47 on dist-test-slave-03cd"
server_key: "edbfda782f480b2b849ddd4884bec22b"
server_key_iv: "94102e1681d186fe740f21f1d4a855ba"
server_key_version: "encryptionkey@0"
I20260619 16:41:48.706619 9401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:41:48.736816 9401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:41:48.740187 9401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:41:48.740429 9401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:41:48.741093 9401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:41:48.741950 9401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:41:48.742022 9401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:48.742084 9401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:41:48.742136 9401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:41:48.751909 9401 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36011
I20260619 16:41:48.752087 9529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36011 every 8 connection(s)
I20260619 16:41:48.753073 9401 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:41:48.763300 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 9401
Jun 19 16:41:48 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:48 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 16:41:48.766552 9267 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:48.755324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38471 (local address 127.0.98.62:39789)
0619 16:41:48.755497 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:48.755502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:48.756570 (+ 1068us) server_negotiation.cc:407] Connection header received
0619 16:41:48.758164 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:48.758168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:48.758235 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:48.758342 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:48.759878 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.760420 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.761648 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.761885 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.762022 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:48.766420 (+ 4398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38471: BlockingRecv error: recv got EOF from 127.0.98.3:38471 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:41:48.767270 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:48.755740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.3:38471)
0619 16:41:48.756246 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:48.756291 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:48.757894 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:48.758587 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:48.758595 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:48.759089 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:48.759708 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:48.759730 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.760560 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.760564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:48.761509 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:48.761519 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.762055 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.762060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:48.762265 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:48.763101 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:48.763139 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:48.766280 (+ 3141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":122,"threads_started":1}
W20260619 16:41:48.767863 9530 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39789 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:39789: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 16:41:48 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:48 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:48.775645 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:48.764448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38342 (local address 127.0.98.62:39789)
0619 16:41:48.764719 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:48.764722 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:48.764736 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:41:48.765047 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:48.765050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:48.765107 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:48.765235 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:48.765995 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.766757 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.767805 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.768138 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.768286 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:48.770034 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:41:48.770040 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:41:48.770059 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:41:48.770087 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:41:48.772846 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:48.773495 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:48.773504 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:48.773512 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:48.773741 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:41:48.774162 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:41:48.774164 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:41:48.774166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:41:48.774654 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:41:48.774811 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 16:41:48.775216 (+ 405us) server_negotiation.cc:299] Negotiation successful
0619 16:41:48.775503 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":95,"threads_started":1}
I20260619 16:41:48.775981 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:48.768395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.3:54531)
0619 16:41:48.768606 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:48.768619 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:48.768753 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:48.769214 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:48.769216 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:48.769371 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:48.769585 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:48.769592 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.770563 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.770570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:48.771634 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:48.771647 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.772646 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.772649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:48.772772 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:48.773287 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:48.773304 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:48.775850 (+ 2546us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141}
W20260619 16:41:48.776010 9267 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:48.768464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54531 (local address 127.0.98.62:39789)
0619 16:41:48.768613 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:48.768619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:48.768687 (+ 68us) server_negotiation.cc:407] Connection header received
0619 16:41:48.768992 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:48.768995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:48.769052 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:48.769146 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:48.769715 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.770306 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.772061 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.772517 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.772649 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:48.775873 (+ 3224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54531: BlockingRecv error: recv got EOF from 127.0.98.3:54531 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 16:41:48 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:48 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:48.782465 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:48.776544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.3:36205)
0619 16:41:48.776667 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:48.776681 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:48.776759 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:48.777018 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:48.777021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:48.777158 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:48.777235 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:48.777240 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.778191 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.778195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:48.778910 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:48.778920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.779418 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.779422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:48.779501 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:48.780208 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:48.780222 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:48.782347 (+ 2125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 16:41:48.782694 9267 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:48.776609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36205 (local address 127.0.98.62:39789)
0619 16:41:48.776742 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:48.776746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:48.776761 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:41:48.776841 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:48.776844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:48.776884 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:48.776960 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:48.777351 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.778014 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.779048 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:48.779269 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:48.779359 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:48.782563 (+ 3204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36205: BlockingRecv error: recv got EOF from 127.0.98.3:36205 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 16:41:48.782755 9530 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 16:41:49 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:49 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:49.373824 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:49.366710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.1:47493)
0619 16:41:49.367002 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:49.367029 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:49.367138 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:49.367506 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:49.367508 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:49.367782 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:49.367911 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:49.367919 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:49.368818 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:49.368821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:49.369462 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:49.369473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:49.370038 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:49.370041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:49.370123 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:49.370870 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:49.370891 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:49.373590 (+ 2699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":101,"threads_started":1}
W20260619 16:41:49.373903 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:49.366815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47493 (local address 127.0.98.62:39789)
0619 16:41:49.367191 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:49.367195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:49.367214 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:41:49.367269 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:49.367271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:49.367336 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:49.367456 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:49.368089 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:49.368688 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:49.369599 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:49.369894 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:49.369994 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:49.373752 (+ 3758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47493: BlockingRecv error: recv got EOF from 127.0.98.1:47493 (error 108)
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":156,"threads_started":1}
Jun 19 16:41:49 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:49 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:49.578584 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:49.570875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.2:56233)
0619 16:41:49.571322 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:49.571346 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:49.571547 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:49.572133 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:49.572136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:49.572401 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:49.572657 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:49.572669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:49.573679 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:49.573683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:49.574395 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:49.574405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:49.574952 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:49.574964 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:49.575045 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:49.575789 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:49.575803 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:49.578398 (+ 2595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":142,"threads_started":1}
W20260619 16:41:49.578867 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:49.571065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56233 (local address 127.0.98.62:39789)
0619 16:41:49.571335 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:49.571339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:49.571460 (+ 121us) server_negotiation.cc:407] Connection header received
0619 16:41:49.571861 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:49.571868 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:49.571948 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:49.572054 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:49.572808 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:49.573470 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:49.574559 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:49.574809 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:49.574901 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:49.578675 (+ 3774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56233: BlockingRecv error: recv got EOF from 127.0.98.2:56233 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Jun 19 16:41:49 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:49 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:49.791242 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:49.783798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.3:60301)
0619 16:41:49.784134 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:49.784161 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:49.784307 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:49.784654 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:49.784658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:49.784912 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:49.785064 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:49.785072 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:49.785969 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:49.785974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:49.786593 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:49.786605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:49.787132 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:49.787135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:49.787229 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:49.788194 (+ 965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:49.788219 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:49.791067 (+ 2848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":126,"threads_started":1}
W20260619 16:41:49.791502 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:49.783931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60301 (local address 127.0.98.62:39789)
0619 16:41:49.784117 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:49.784120 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:49.784309 (+ 189us) server_negotiation.cc:407] Connection header received
0619 16:41:49.784383 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:49.784386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:49.784442 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:49.784553 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:49.785235 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:49.785776 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:49.786757 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:49.786992 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:49.787109 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:49.791350 (+ 4241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60301: BlockingRecv error: recv got EOF from 127.0.98.3:60301 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 16:41:50 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:50 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:50.381158 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:50.375070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.1:53903)
0619 16:41:50.375397 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:50.375417 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:50.375545 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:50.375931 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:50.375933 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:50.376120 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:50.376221 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:50.376226 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:50.377044 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:50.377047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:50.377610 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:50.377618 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:50.378157 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:50.378161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:50.378252 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:50.378893 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:50.378910 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:50.381000 (+ 2090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":79,"threads_started":1}
W20260619 16:41:50.381453 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:50.375106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53903 (local address 127.0.98.62:39789)
0619 16:41:50.375456 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:50.375460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:50.375475 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:41:50.375710 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:50.375723 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:50.375785 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:50.375876 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:50.376363 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:50.376918 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:50.377749 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:50.377993 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:50.378090 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:50.381193 (+ 3103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53903: BlockingRecv error: recv got EOF from 127.0.98.1:53903 (error 108)
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":101,"threads_started":1}
Jun 19 16:41:50 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:50 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:50.585959 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:50.579786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.2:44965)
0619 16:41:50.580135 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:50.580148 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:50.580304 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:50.580616 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:50.580619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:50.580852 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:50.580958 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:50.580965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:50.581754 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:50.581759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:50.582328 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:50.582335 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:50.582800 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:50.582803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:50.582889 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:50.583486 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:50.583498 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:50.585793 (+ 2295us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":99,"threads_started":1}
W20260619 16:41:50.586073 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:50.579830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44965 (local address 127.0.98.62:39789)
0619 16:41:50.580044 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:50.580050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:50.580235 (+ 185us) server_negotiation.cc:407] Connection header received
0619 16:41:50.580415 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:50.580418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:50.580480 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:50.580582 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:50.581096 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:50.581638 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:50.582462 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:50.582684 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:50.582795 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:50.585933 (+ 3138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44965: BlockingRecv error: recv got EOF from 127.0.98.2:44965 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 16:41:50 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:50 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:50.798537 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:50.792352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.3:35213)
0619 16:41:50.792668 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:50.792686 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:50.792852 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:50.793149 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:50.793152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:50.793378 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:50.793478 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:50.793483 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:50.794306 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:50.794311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:50.794962 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:50.794971 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:50.795435 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:50.795438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:50.795521 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:50.796161 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:50.796177 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:50.798330 (+ 2153us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":110,"threads_started":1}
W20260619 16:41:50.798584 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:50.792517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35213 (local address 127.0.98.62:39789)
0619 16:41:50.792667 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:50.792671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:50.792768 (+ 97us) server_negotiation.cc:407] Connection header received
0619 16:41:50.792955 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:50.792959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:50.793025 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:50.793126 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:50.793661 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:50.794189 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:50.795092 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:50.795317 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:50.795432 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:50.798476 (+ 3044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35213: BlockingRecv error: recv got EOF from 127.0.98.3:35213 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 16:41:51 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:51 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:51.390177 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:51.382368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.1:33917)
0619 16:41:51.382756 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:51.382782 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:51.382949 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:51.383443 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:51.383446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:51.383685 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:51.383873 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:51.383881 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:51.384877 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:51.384883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:51.385693 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:51.385707 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:51.386497 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:51.386504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:51.386635 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:51.387415 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:51.387433 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:51.389989 (+ 2556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":101,"threads_started":1}
W20260619 16:41:51.390453 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:51.382497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33917 (local address 127.0.98.62:39789)
0619 16:41:51.382935 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:51.383021 (+ 86us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:51.383060 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:41:51.383131 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:51.383135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:51.383207 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:51.383360 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:51.384043 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:51.384675 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:51.385928 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:51.386320 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:51.386452 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:51.390272 (+ 3820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33917: BlockingRecv error: recv got EOF from 127.0.98.1:33917 (error 108)
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":112,"threads_started":1}
Jun 19 16:41:51 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:51 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:51.593374 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:51.587190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.2:39853)
0619 16:41:51.587538 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:51.587552 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:51.587685 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:51.587993 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:51.587995 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:51.588189 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:51.588298 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:51.588304 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:51.589251 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:51.589254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:51.589851 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:51.589860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:51.590472 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:51.590474 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:51.590551 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:51.591189 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:51.591202 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:51.593224 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":119,"threads_started":1}
W20260619 16:41:51.593803 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:51.587231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39853 (local address 127.0.98.62:39789)
0619 16:41:51.587437 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:51.587442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:51.587621 (+ 179us) server_negotiation.cc:407] Connection header received
0619 16:41:51.587788 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:51.587790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:51.587862 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:51.587941 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:51.588441 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:51.589068 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:51.590032 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:51.590330 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:51.590448 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:51.593627 (+ 3179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39853: BlockingRecv error: recv got EOF from 127.0.98.2:39853 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 16:41:51 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:51 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:51.806864 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:51.799912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.3:59921)
0619 16:41:51.800402 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:51.800428 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:51.800646 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:51.801122 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:51.801124 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:51.801395 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:51.801511 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:51.801517 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:51.802410 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:51.802413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:51.803016 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:51.803025 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:51.803647 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:51.803650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:51.803838 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:51.804545 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:51.804558 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:51.806709 (+ 2151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":128,"threads_started":1}
W20260619 16:41:51.807418 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:51.799952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59921 (local address 127.0.98.62:39789)
0619 16:41:51.800145 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:51.800150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:51.800556 (+ 406us) server_negotiation.cc:407] Connection header received
0619 16:41:51.800803 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:51.800806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:51.800867 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:51.801060 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:51.801640 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:51.802258 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:51.803186 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:51.803453 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:51.803569 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:51.807238 (+ 3669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59921: BlockingRecv error: recv got EOF from 127.0.98.3:59921 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 16:41:52 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:52 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:52.400954 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:52.391463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.1:55227)
0619 16:41:52.391981 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:52.391999 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:52.392195 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:52.392733 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:52.392736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:52.392971 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:52.393120 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:52.393126 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:52.394225 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:52.394228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:52.394864 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:52.394876 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:52.395516 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:52.395520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:52.395610 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:52.396412 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:52.396428 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:52.400728 (+ 4300us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":155,"threads_started":1}
W20260619 16:41:52.401070 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:52.391698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55227 (local address 127.0.98.62:39789)
0619 16:41:52.392018 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:52.392022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:52.392135 (+ 113us) server_negotiation.cc:407] Connection header received
0619 16:41:52.392464 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:52.392467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:52.392527 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:52.392634 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:52.393324 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:52.394037 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:52.395078 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:52.395327 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:52.395438 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:52.400908 (+ 5470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55227: BlockingRecv error: recv got EOF from 127.0.98.1:55227 (error 108)
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":67,"threads_started":1}
Jun 19 16:41:52 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:52 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:52.602341 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:52.594560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.2:47091)
0619 16:41:52.595014 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:52.595063 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:52.595256 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:52.595604 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:52.595607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:52.595871 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:52.595972 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:52.595978 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:52.596841 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:52.596846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:52.597682 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:52.597692 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:52.598416 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:52.598422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:52.598585 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:52.599472 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:52.599487 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:52.602168 (+ 2681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":119,"threads_started":1}
W20260619 16:41:52.602730 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:52.594754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47091 (local address 127.0.98.62:39789)
0619 16:41:52.594993 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:52.595000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:52.595188 (+ 188us) server_negotiation.cc:407] Connection header received
0619 16:41:52.595404 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:52.595408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:52.595472 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:52.595583 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:52.596110 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:52.596712 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:52.597813 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:52.598153 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:52.598276 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:52.602438 (+ 4162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47091: BlockingRecv error: recv got EOF from 127.0.98.2:47091 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Jun 19 16:41:52 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:52 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:52.814473 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:52.807907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.3:43701)
0619 16:41:52.808214 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:52.808228 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:52.808339 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:52.808728 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:52.808731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:52.808915 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:52.809025 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:52.809031 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:52.809950 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:52.809954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:52.810911 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:52.810922 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:52.811707 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:52.811713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:52.811795 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:52.812429 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:52.812441 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:52.814332 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":113,"threads_started":1}
W20260619 16:41:52.814908 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:52.808013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43701 (local address 127.0.98.62:39789)
0619 16:41:52.808218 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:52.808222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:52.808312 (+ 90us) server_negotiation.cc:407] Connection header received
0619 16:41:52.808493 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:52.808496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:52.808571 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:52.808675 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:52.809186 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:52.809794 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:52.811159 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:52.811446 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:52.811579 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:52.814735 (+ 3156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43701: BlockingRecv error: recv got EOF from 127.0.98.3:43701 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 19 16:41:53 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:53 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:53.410070 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:53.402652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.1:55827)
0619 16:41:53.403126 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:53.403144 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:53.403295 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:53.403850 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:53.403856 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:53.404190 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:53.404366 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:53.404379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:53.405341 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:53.405345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:53.406232 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:53.406244 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:53.406777 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:53.406779 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:53.406856 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:53.407492 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:53.407506 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:53.409873 (+ 2367us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":122,"threads_started":1}
W20260619 16:41:53.410234 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:53.402918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55827 (local address 127.0.98.62:39789)
0619 16:41:53.403203 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:53.403208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:53.403226 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:41:53.403421 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:53.403424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:53.403516 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:53.403624 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:53.404515 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:53.405141 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:53.406367 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:53.406615 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:53.406710 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:53.410085 (+ 3375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55827: BlockingRecv error: recv got EOF from 127.0.98.1:55827 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":99,"threads_started":1}
Jun 19 16:41:53 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:53 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:53.617583 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:53.603744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.2:40823)
0619 16:41:53.604100 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:53.604117 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:53.604250 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:53.604678 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:53.604682 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:53.605129 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:53.605283 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:53.605291 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:53.606552 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:53.606555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:53.607399 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:53.607410 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:53.608047 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:53.608053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:53.608157 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:53.613866 (+ 5709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:53.613886 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:53.617397 (+ 3511us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":125,"threads_started":1}
W20260619 16:41:53.617843 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:53.603890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40823 (local address 127.0.98.62:39789)
0619 16:41:53.604086 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:53.604090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:53.604214 (+ 124us) server_negotiation.cc:407] Connection header received
0619 16:41:53.604359 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:53.604362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:53.604420 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:53.604546 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:53.605501 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:53.606315 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:53.607567 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:53.607909 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:53.608014 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:53.617664 (+ 9650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40823: BlockingRecv error: recv got EOF from 127.0.98.2:40823 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 19 16:41:53 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:53 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:53.822731 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:53.815585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.3:58849)
0619 16:41:53.816023 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:53.816041 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:53.816200 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:53.816545 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:53.816548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:53.816780 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:53.816925 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:53.816931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:53.817873 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:53.817876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:53.818541 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:53.818551 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:53.819094 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:53.819097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:53.819175 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:53.819949 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:53.819962 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:53.822552 (+ 2590us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":162,"threads_started":1}
W20260619 16:41:53.822856 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:53.815830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58849 (local address 127.0.98.62:39789)
0619 16:41:53.816045 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:53.816049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:53.816198 (+ 149us) server_negotiation.cc:407] Connection header received
0619 16:41:53.816298 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:53.816301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:53.816358 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:53.816455 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:53.817123 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:53.817699 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:53.818684 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:53.818915 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:53.819015 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:53.822736 (+ 3721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58849: BlockingRecv error: recv got EOF from 127.0.98.3:58849 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jun 19 16:41:54 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:54 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:54.418385 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:54.411321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.1:33575)
0619 16:41:54.411743 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:54.411758 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:54.411900 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:54.412200 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:54.412202 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:54.412419 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:54.412554 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:54.412559 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:54.413349 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:54.413351 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:54.414195 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:54.414211 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:54.414711 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:54.414713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:54.414787 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:54.415580 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:54.415595 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:54.418168 (+ 2573us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":128,"threads_started":1}
W20260619 16:41:54.418684 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:54.411396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33575 (local address 127.0.98.62:39789)
0619 16:41:54.411708 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:54.411712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:54.411856 (+ 144us) server_negotiation.cc:407] Connection header received
0619 16:41:54.412038 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:54.412040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:54.412094 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:54.412180 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:54.412669 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:54.413208 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:54.414331 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:54.414588 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:54.414677 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:54.418485 (+ 3808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33575: BlockingRecv error: recv got EOF from 127.0.98.1:33575 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":110,"threads_started":1}
Jun 19 16:41:54 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:54 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:54.626670 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:54.618892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.2:57503)
0619 16:41:54.619240 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:54.619255 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:54.619394 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:54.619876 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:54.619879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:54.620099 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:54.620294 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:54.620300 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:54.621471 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:54.621477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:54.622336 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:54.622346 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:54.623088 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:54.623095 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:54.623228 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:54.624018 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:54.624032 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:54.626463 (+ 2431us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":123,"threads_started":1}
W20260619 16:41:54.626802 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:54.619118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57503 (local address 127.0.98.62:39789)
0619 16:41:54.619285 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:54.619289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:54.619328 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:41:54.619594 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:54.619597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:54.619686 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:54.619801 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:54.620488 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:54.621295 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:54.622576 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:54.622914 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:54.623042 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:54.626682 (+ 3640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57503: BlockingRecv error: recv got EOF from 127.0.98.2:57503 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 16:41:54 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:54 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:54.830595 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:54.823921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.3:51181)
0619 16:41:54.824304 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:54.824319 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:54.824446 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:54.824806 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:54.824808 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:54.825024 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:54.825150 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:54.825156 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:54.826005 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:54.826008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:54.826670 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:54.826680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:54.827178 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:54.827181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:54.827321 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:54.828095 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:54.828108 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:54.830436 (+ 2328us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":157,"threads_started":1}
W20260619 16:41:54.830853 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:54.824070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51181 (local address 127.0.98.62:39789)
0619 16:41:54.824246 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:54.824250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:54.824413 (+ 163us) server_negotiation.cc:407] Connection header received
0619 16:41:54.824590 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:54.824593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:54.824648 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:54.824745 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:54.825304 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:54.825878 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:54.826799 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:54.827027 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:54.827117 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:54.830690 (+ 3573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51181: BlockingRecv error: recv got EOF from 127.0.98.3:51181 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 19 16:41:55 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:55 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:55.426504 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:55.419412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.1:34807)
0619 16:41:55.419836 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:55.419854 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:55.419971 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:55.420307 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:55.420310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:55.420588 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:55.420716 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:55.420723 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:55.421575 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:55.421579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:55.422218 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:55.422229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:55.422991 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:55.422995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:55.423072 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:55.423872 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:55.423885 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:55.426336 (+ 2451us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":83,"threads_started":1}
W20260619 16:41:55.426890 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:55.419476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34807 (local address 127.0.98.62:39789)
0619 16:41:55.419916 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:55.419920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:55.419938 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:41:55.420080 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:55.420082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:55.420131 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:55.420221 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:55.420842 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:55.421392 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:55.422382 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:55.422816 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:55.422947 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:55.426648 (+ 3701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34807: BlockingRecv error: recv got EOF from 127.0.98.1:34807 (error 108)
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":193,"threads_started":1}
Jun 19 16:41:55 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:55 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:55.634855 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:55.628012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.2:60851)
0619 16:41:55.628429 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:55.628444 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:55.628573 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:55.628857 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:55.628859 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:55.629075 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:55.629199 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:55.629204 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:55.630011 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:55.630014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:55.630774 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:55.630787 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:55.631318 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:55.631322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:55.631437 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:55.632235 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:55.632257 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:55.634692 (+ 2435us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":132,"threads_started":1}
W20260619 16:41:55.635001 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:55.628173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60851 (local address 127.0.98.62:39789)
0619 16:41:55.628357 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:55.628361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:55.628591 (+ 230us) server_negotiation.cc:407] Connection header received
0619 16:41:55.628656 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:55.628658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:55.628711 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:55.628814 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:55.629319 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:55.629870 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:55.630926 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:55.631162 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:55.631305 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:55.634846 (+ 3541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60851: BlockingRecv error: recv got EOF from 127.0.98.2:60851 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 19 16:41:55 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:55 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:55.838056 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:55.831750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.3:46995)
0619 16:41:55.832172 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:55.832191 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:55.832324 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:55.832689 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:55.832692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:55.832928 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:55.833057 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:55.833063 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:55.833826 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:55.833829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:55.834464 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:55.834474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:55.834918 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:55.834921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:55.834987 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:55.835738 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:55.835756 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:55.837895 (+ 2139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":134,"threads_started":1}
W20260619 16:41:55.838326 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:55.831843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46995 (local address 127.0.98.62:39789)
0619 16:41:55.831993 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:55.831996 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:55.832280 (+ 284us) server_negotiation.cc:407] Connection header received
0619 16:41:55.832413 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:55.832415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:55.832463 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:55.832569 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:55.833174 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:55.833699 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:55.834579 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:55.834803 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:55.834888 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:55.838154 (+ 3266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46995: BlockingRecv error: recv got EOF from 127.0.98.3:46995 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 16:41:56 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:56 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:56.434188 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:56.427621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.1:36359)
0619 16:41:56.428019 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:56.428036 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:56.428146 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:56.428549 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:56.428551 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:56.428769 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:56.428924 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:56.428931 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:56.429884 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:56.429887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:56.430564 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:56.430573 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:56.431124 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:56.431127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:56.431198 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:56.431954 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:56.431967 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:56.434000 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":143,"threads_started":1}
W20260619 16:41:56.434495 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:56.427923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36359 (local address 127.0.98.62:39789)
0619 16:41:56.428248 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:56.428253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:56.428268 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:41:56.428317 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:56.428319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:56.428370 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:56.428493 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:56.429142 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:56.429737 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:56.430700 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:56.430997 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:56.431091 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:56.434280 (+ 3189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36359: BlockingRecv error: recv got EOF from 127.0.98.1:36359 (error 108)
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":139,"threads_started":1}
Jun 19 16:41:56 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:56 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:56.644656 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:56.636344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.2:37701)
0619 16:41:56.636871 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:56.636885 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:56.637063 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:56.637636 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:56.637640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:56.637873 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:56.638034 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:56.638043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:56.639547 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:56.639551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:56.640308 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:56.640320 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:56.641176 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:56.641179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:56.641258 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:56.642042 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:56.642056 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:56.644413 (+ 2357us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":145,"threads_started":1}
W20260619 16:41:56.644749 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:56.636513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37701 (local address 127.0.98.62:39789)
0619 16:41:56.636733 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:56.636738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:56.636993 (+ 255us) server_negotiation.cc:407] Connection header received
0619 16:41:56.637354 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:56.637359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:56.637464 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:56.637576 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:56.638216 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:56.639314 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:56.640688 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:56.640964 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:56.641104 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:56.644615 (+ 3511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37701: BlockingRecv error: recv got EOF from 127.0.98.2:37701 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 16:41:56 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:56 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:56.846553 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:56.839299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.3:54047)
0619 16:41:56.839757 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:56.839774 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:56.839927 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:56.840361 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:56.840364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:56.840596 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:56.840731 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:56.840737 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:56.841914 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:56.841919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:56.842784 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:56.842797 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:56.843305 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:56.843309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:56.843393 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:56.844102 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:56.844115 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:56.846352 (+ 2237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":145,"threads_started":1}
W20260619 16:41:56.846683 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:56.839521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54047 (local address 127.0.98.62:39789)
0619 16:41:56.839760 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:56.839764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:56.839923 (+ 159us) server_negotiation.cc:407] Connection header received
0619 16:41:56.840029 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:56.840032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:56.840099 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:56.840263 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:56.840953 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:56.841764 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:56.842927 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:56.843173 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:56.843293 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:56.846547 (+ 3254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54047: BlockingRecv error: recv got EOF from 127.0.98.3:54047 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 19 16:41:57 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:57 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:57.443042 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:57.435517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.1:40173)
0619 16:41:57.435888 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:57.435909 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:57.436068 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:57.436573 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:57.436577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:57.436823 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:57.436967 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:57.436974 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:57.437879 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:57.437882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:57.438677 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:57.438690 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:57.439343 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:57.439349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:57.439473 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:57.440341 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:57.440358 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:57.442891 (+ 2533us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":201,"threads_started":1}
W20260619 16:41:57.443311 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:57.435821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40173 (local address 127.0.98.62:39789)
0619 16:41:57.436213 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:57.436217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:57.436236 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:41:57.436294 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:57.436297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:57.436347 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:57.436474 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:57.437145 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:57.437729 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:57.438870 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:57.439144 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:57.439288 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:57.443088 (+ 3800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40173: BlockingRecv error: recv got EOF from 127.0.98.1:40173 (error 108)
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":119,"threads_started":1}
Jun 19 16:41:57 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:57 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:57.652902 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:57.645794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.2:53851)
0619 16:41:57.646135 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:57.646150 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:57.646283 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:57.646700 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:57.646704 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:57.646936 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:57.647079 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:57.647085 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:57.648059 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:57.648061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:57.648756 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:57.648766 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:57.649366 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:57.649369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:57.649445 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:57.650128 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:57.650141 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:57.652687 (+ 2546us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":138,"threads_started":1}
W20260619 16:41:57.653163 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:57.645915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53851 (local address 127.0.98.62:39789)
0619 16:41:57.646141 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:57.646145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:57.646285 (+ 140us) server_negotiation.cc:407] Connection header received
0619 16:41:57.646414 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:57.646417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:57.646480 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:57.646628 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:57.647240 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:57.647913 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:57.648945 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:57.649202 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:57.649311 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:57.652945 (+ 3634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53851: BlockingRecv error: recv got EOF from 127.0.98.2:53851 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 19 16:41:57 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:57 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:57.854835 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:57.847961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.3:39803)
0619 16:41:57.848463 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:57.848479 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:57.848677 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:57.848985 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:57.848988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:57.849240 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:57.849352 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:57.849358 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:57.850217 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:57.850223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:57.850999 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:57.851008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:57.851486 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:57.851496 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:57.851577 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:57.852324 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:57.852369 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:57.854614 (+ 2245us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":283,"threads_started":1}
W20260619 16:41:57.854903 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:57.848131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39803 (local address 127.0.98.62:39789)
0619 16:41:57.848332 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:57.848339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:57.848594 (+ 255us) server_negotiation.cc:407] Connection header received
0619 16:41:57.848798 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:57.848802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:57.848864 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:57.848957 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:57.849480 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:57.850087 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:57.851130 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:57.851369 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:57.851480 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:57.854774 (+ 3294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39803: BlockingRecv error: recv got EOF from 127.0.98.3:39803 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 19 16:41:58 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:58 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:58.450816 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:58.444162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.1:58023)
0619 16:41:58.444530 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:58.444547 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:58.444691 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:58.444979 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:58.444981 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:58.445218 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:58.445375 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:58.445380 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:58.446364 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:58.446367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:58.447045 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:58.447056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:58.447647 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:58.447650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:58.447822 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:58.448532 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:58.448545 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:58.450656 (+ 2111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":107,"threads_started":1}
W20260619 16:41:58.451095 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:58.444225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58023 (local address 127.0.98.62:39789)
0619 16:41:58.444508 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:58.444512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:58.444659 (+ 147us) server_negotiation.cc:407] Connection header received
0619 16:41:58.444787 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:58.444789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:58.444839 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:58.444929 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:58.445547 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:58.446198 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:58.447226 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:58.447492 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:58.447599 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:58.450894 (+ 3295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58023: BlockingRecv error: recv got EOF from 127.0.98.1:58023 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":63,"threads_started":1}
Jun 19 16:41:58 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:58 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:58.664762 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:58.654036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.2:52689)
0619 16:41:58.654321 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:58.654337 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:58.654439 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:58.654782 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:58.654784 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:58.654990 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:58.655099 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:58.655105 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:58.656062 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:58.656064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:58.656953 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:58.656964 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:58.657525 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:58.657528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:58.657597 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:58.661852 (+ 4255us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:58.661876 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:58.664617 (+ 2741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":91,"threads_started":1}
W20260619 16:41:58.664973 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:58.654185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52689 (local address 127.0.98.62:39789)
0619 16:41:58.654376 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:58.654380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:58.654403 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:41:58.654547 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:58.654549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:58.654618 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:58.654727 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:58.655254 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:58.655908 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:58.657136 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:58.657390 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:58.657499 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:58.664808 (+ 7309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52689: BlockingRecv error: recv got EOF from 127.0.98.2:52689 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 19 16:41:58 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:58 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:58.863564 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:58.855908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.3:46851)
0619 16:41:58.856368 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:58.856390 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:58.856548 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:58.856848 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:58.856851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:58.857098 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:58.857198 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:58.857204 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:58.858116 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:58.858121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:58.858762 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:58.858771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:58.859247 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:58.859250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:58.859344 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:58.860417 (+ 1073us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:58.860432 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:58.863376 (+ 2944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":117,"threads_started":1}
W20260619 16:41:58.863934 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:58.856125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46851 (local address 127.0.98.62:39789)
0619 16:41:58.856325 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:58.856330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:58.856488 (+ 158us) server_negotiation.cc:407] Connection header received
0619 16:41:58.856651 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:58.856654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:58.856708 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:58.856828 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:58.857375 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:58.857986 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:58.858877 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:58.859122 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:58.859237 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:58.863756 (+ 4519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46851: BlockingRecv error: recv got EOF from 127.0.98.3:46851 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Jun 19 16:41:59 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:59 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:59.459282 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:59.452138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.1:37199)
0619 16:41:59.452532 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:59.452548 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:59.452694 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:59.453102 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:59.453105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:59.453330 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:59.453462 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:59.453468 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:59.454376 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:59.454379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:59.455029 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:59.455039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:59.455892 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:59.455898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:59.455984 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:59.456680 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:59.456691 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:59.459106 (+ 2415us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":184,"threads_started":1}
W20260619 16:41:59.459591 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:59.452192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37199 (local address 127.0.98.62:39789)
0619 16:41:59.452546 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:59.452552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:59.452631 (+ 79us) server_negotiation.cc:407] Connection header received
0619 16:41:59.452822 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:59.452825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:59.452885 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:59.452986 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:59.453609 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:59.454220 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:59.455229 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:59.455716 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:59.455892 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:59.459408 (+ 3516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37199: BlockingRecv error: recv got EOF from 127.0.98.1:37199 (error 108)
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":85,"threads_started":1}
Jun 19 16:41:59 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:59 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:59.672968 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:59.666042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.2:51495)
0619 16:41:59.666346 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:59.666361 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:59.666461 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:59.666897 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:59.666900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:59.667089 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:59.667201 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:59.667206 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:59.668172 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:59.668176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:59.668808 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:59.668818 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:59.669356 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:59.669358 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:59.669423 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:59.670199 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:59.670214 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:59.672811 (+ 2597us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":93,"threads_started":1}
W20260619 16:41:59.673228 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:59.666196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51495 (local address 127.0.98.62:39789)
0619 16:41:59.666407 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:59.666411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:59.666437 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:41:59.666620 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:59.666624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:59.666688 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:59.666808 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:59.667350 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:59.668026 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:59.668963 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:59.669231 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:59.669334 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:59.673061 (+ 3727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51495: BlockingRecv error: recv got EOF from 127.0.98.2:51495 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Jun 19 16:41:59 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:41:59 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:41:59.870358 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:41:59.864691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.3:37733)
0619 16:41:59.865088 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0619 16:41:59.865102 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:41:59.865280 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:41:59.865563 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:41:59.865566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:41:59.865778 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:41:59.865867 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:59.865873 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:59.866643 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:59.866649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:59.867193 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:41:59.867200 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:41:59.867697 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:41:59.867702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:41:59.867787 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:59.868352 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:41:59.868364 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:41:59.870153 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":110,"threads_started":1}
W20260619 16:41:59.870431 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:41:59.864815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37733 (local address 127.0.98.62:39789)
0619 16:41:59.864974 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 16:41:59.864979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:41:59.865197 (+ 218us) server_negotiation.cc:407] Connection header received
0619 16:41:59.865395 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:41:59.865398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:41:59.865452 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:41:59.865541 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:41:59.865984 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:59.866541 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:59.867316 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:41:59.867541 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:41:59.867648 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:41:59.870298 (+ 2650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37733: BlockingRecv error: recv got EOF from 127.0.98.3:37733 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 16:42:00 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:42:00 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:42:00.466734 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:00.460273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.1:54185)
0619 16:42:00.460830 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0619 16:42:00.460847 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:42:00.460982 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:42:00.461329 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:42:00.461332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:42:00.461562 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:42:00.461692 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:00.461700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:00.462472 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:00.462479 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:00.463038 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:00.463046 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:00.463512 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:00.463515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:00.463597 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:00.464310 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:42:00.464322 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:42:00.466563 (+ 2241us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":116,"threads_started":1}
W20260619 16:42:00.466935 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:42:00.460337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54185 (local address 127.0.98.62:39789)
0619 16:42:00.460717 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:00.460720 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:00.460966 (+ 246us) server_negotiation.cc:407] Connection header received
0619 16:42:00.461115 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:00.461117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:00.461170 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:00.461292 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:42:00.461810 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:00.462359 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:00.463157 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:00.463396 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:00.463504 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:00.466737 (+ 3233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54185: BlockingRecv error: recv got EOF from 127.0.98.1:54185 (error 108)
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":132,"threads_started":1}
Jun 19 16:42:00 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:42:00 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:42:00.680151 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:00.673921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.2:49175)
0619 16:42:00.674290 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0619 16:42:00.674307 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:42:00.674470 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:42:00.674740 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:42:00.674743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:42:00.674961 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:42:00.675054 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:00.675059 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:00.675880 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:00.675885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:00.676470 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:00.676482 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:00.676969 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:00.676972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:00.677055 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:00.677773 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:42:00.677785 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:42:00.679997 (+ 2212us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":120,"threads_started":1}
W20260619 16:42:00.680498 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:42:00.674035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49175 (local address 127.0.98.62:39789)
0619 16:42:00.674218 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:00.674223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:00.674404 (+ 181us) server_negotiation.cc:407] Connection header received
0619 16:42:00.674572 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:00.674576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:00.674626 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:00.674716 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:42:00.675196 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:00.675775 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:00.676618 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:00.676849 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:00.676957 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:00.680231 (+ 3274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49175: BlockingRecv error: recv got EOF from 127.0.98.2:49175 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 16:42:00 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:42:00 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:42:00.877470 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:00.871397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.3:35611)
0619 16:42:00.871779 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0619 16:42:00.871798 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:42:00.871956 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:42:00.872233 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:42:00.872236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:42:00.872451 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:42:00.872539 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:00.872544 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:00.873286 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:00.873291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:00.873839 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:00.873846 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:00.874329 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:00.874333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:00.874416 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:00.875035 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:42:00.875047 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:42:00.877285 (+ 2238us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":119,"threads_started":1}
W20260619 16:42:00.877621 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:42:00.871458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35611 (local address 127.0.98.62:39789)
0619 16:42:00.871630 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:00.871635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:00.871893 (+ 258us) server_negotiation.cc:407] Connection header received
0619 16:42:00.872052 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:00.872056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:00.872119 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:00.872206 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:42:00.872654 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:00.873179 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:00.873958 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:00.874201 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:00.874329 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:00.877457 (+ 3128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35611: BlockingRecv error: recv got EOF from 127.0.98.3:35611 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 16:42:01 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:42:01 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:42:01.475265 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:01.468009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.1:39041)
0619 16:42:01.468379 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0619 16:42:01.468394 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:42:01.468510 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:42:01.468963 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:42:01.468967 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:42:01.469231 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:42:01.469372 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:01.469378 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:01.470280 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:01.470284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:01.471017 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:01.471028 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:01.471563 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:01.471566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:01.471722 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:01.472414 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:42:01.472428 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:42:01.475081 (+ 2653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":141,"threads_started":1}
W20260619 16:42:01.475464 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:42:01.468157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39041 (local address 127.0.98.62:39789)
0619 16:42:01.468441 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:01.468445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:01.468464 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:42:01.468629 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:01.468632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:01.468692 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:01.468828 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:42:01.469534 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:01.470155 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:01.471167 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:01.471421 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:01.471525 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:01.475286 (+ 3761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39041: BlockingRecv error: recv got EOF from 127.0.98.1:39041 (error 108)
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":103,"threads_started":1}
Jun 19 16:42:01 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:42:01 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:42:01.688813 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:01.681306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.2:42899)
0619 16:42:01.681799 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0619 16:42:01.681821 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:42:01.681999 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:42:01.682414 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:42:01.682418 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:42:01.682724 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:42:01.682889 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:01.682896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:01.683925 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:01.683928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:01.684626 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:01.684637 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:01.685442 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:01.685445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:01.685518 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:01.686235 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:42:01.686248 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:42:01.688651 (+ 2403us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":151,"threads_started":1}
W20260619 16:42:01.689096 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:42:01.681558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42899 (local address 127.0.98.62:39789)
0619 16:42:01.681799 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:01.681806 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:01.681927 (+ 121us) server_negotiation.cc:407] Connection header received
0619 16:42:01.682136 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:01.682140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:01.682225 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:01.682325 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:42:01.683038 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:01.683797 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:01.684766 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:01.685292 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:01.685407 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:01.688926 (+ 3519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42899: BlockingRecv error: recv got EOF from 127.0.98.2:42899 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
Jun 19 16:42:01 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:42:01 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:42:01.885103 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:01.878561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.3:45363)
0619 16:42:01.878856 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 16:42:01.878871 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:42:01.878982 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:42:01.879389 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:42:01.879391 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:42:01.879589 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:42:01.879728 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:01.879733 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:01.880593 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:01.880596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:01.881185 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:01.881193 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:01.881726 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:01.881729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:01.881801 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:01.882443 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:42:01.882454 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:42:01.884898 (+ 2444us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":99,"threads_started":1}
W20260619 16:42:01.885218 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:42:01.878693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45363 (local address 127.0.98.62:39789)
0619 16:42:01.878889 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:01.878894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:01.878921 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:42:01.879154 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:01.879157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:01.879223 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:01.879352 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:42:01.879857 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:01.880475 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:01.881323 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:01.881572 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:01.881672 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:01.885097 (+ 3425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45363: BlockingRecv error: recv got EOF from 127.0.98.3:45363 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 19 16:42:02 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:42:02 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:42:02.483037 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:02.476401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.1:49157)
0619 16:42:02.476712 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 16:42:02.476726 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:42:02.476826 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:42:02.477266 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:42:02.477268 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:42:02.477554 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:42:02.477686 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:02.477693 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:02.478944 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:02.478948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:02.479558 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:02.479566 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:02.480115 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:02.480118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:02.480188 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:02.480857 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:42:02.480869 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:42:02.482879 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":103,"threads_started":1}
W20260619 16:42:02.483279 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:42:02.476492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49157 (local address 127.0.98.62:39789)
0619 16:42:02.476889 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:02.476894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:02.476913 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:42:02.476968 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:02.476970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:02.477076 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:02.477236 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:42:02.477960 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:02.478808 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:02.479723 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:02.479977 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:02.480100 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:02.483093 (+ 2993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49157: BlockingRecv error: recv got EOF from 127.0.98.1:49157 (error 108)
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":115,"threads_started":1}
Jun 19 16:42:02 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:42:02 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:42:02.696206 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:02.689721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.2:47365)
0619 16:42:02.690084 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0619 16:42:02.690100 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:42:02.690240 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:42:02.690654 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:42:02.690656 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:42:02.690848 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:42:02.690970 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:02.690976 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:02.691994 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:02.691997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:02.692627 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:02.692636 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:02.693188 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:02.693190 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:02.693257 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:02.693879 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:42:02.693893 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:42:02.696049 (+ 2156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":151,"threads_started":1}
W20260619 16:42:02.696450 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:42:02.689790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47365 (local address 127.0.98.62:39789)
0619 16:42:02.689982 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:02.689987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:02.690184 (+ 197us) server_negotiation.cc:407] Connection header received
0619 16:42:02.690398 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:02.690401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:02.690476 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:02.690602 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:42:02.691113 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:02.691872 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:02.692757 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:02.693071 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:02.693193 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:02.696249 (+ 3056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47365: BlockingRecv error: recv got EOF from 127.0.98.2:47365 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 19 16:42:02 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:42:02 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:42:02.893533 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:02.886363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.3:37465)
0619 16:42:02.886884 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0619 16:42:02.886901 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:42:02.887087 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:42:02.887468 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:42:02.887471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:42:02.887792 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:42:02.887919 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:02.887925 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:02.888923 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:02.888926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:02.889566 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:02.889575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:02.890154 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:02.890157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:02.890230 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:02.890925 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:42:02.890938 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:42:02.893359 (+ 2421us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":173,"threads_started":1}
W20260619 16:42:02.893811 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:42:02.886563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37465 (local address 127.0.98.62:39789)
0619 16:42:02.886810 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:02.886816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:02.887011 (+ 195us) server_negotiation.cc:407] Connection header received
0619 16:42:02.887246 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:02.887249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:02.887315 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:02.887433 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:42:02.888049 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:02.888773 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:02.889733 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:02.890019 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:02.890204 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:02.893590 (+ 3386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37465: BlockingRecv error: recv got EOF from 127.0.98.3:37465 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Jun 19 16:42:03 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:42:03 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:42:03.491266 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:03.484241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.1:45073)
0619 16:42:03.484681 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0619 16:42:03.484700 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:42:03.484928 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:42:03.485600 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:42:03.485603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:42:03.485884 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:42:03.486024 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:03.486035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:03.486828 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:03.486833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:03.487415 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:03.487423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:03.487954 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:03.487957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:03.488038 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:03.488672 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:42:03.488688 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:42:03.491104 (+ 2416us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":146,"threads_started":1}
W20260619 16:42:03.491504 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:42:03.484319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45073 (local address 127.0.98.62:39789)
0619 16:42:03.484702 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:03.484710 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:03.485274 (+ 564us) server_negotiation.cc:407] Connection header received
0619 16:42:03.485359 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:03.485363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:03.485453 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:03.485585 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:42:03.486168 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:03.486716 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:03.487534 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:03.487835 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:03.487944 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:03.491298 (+ 3354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45073: BlockingRecv error: recv got EOF from 127.0.98.1:45073 (error 108)
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":109,"threads_started":1}
Jun 19 16:42:03 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:42:03 dist-test-slave-03cd krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:42:03.707285 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:03.697392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39789 (local address 127.0.98.2:39271)
0619 16:42:03.697782 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0619 16:42:03.697797 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:42:03.697952 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:42:03.698225 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:42:03.698228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:42:03.698438 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:42:03.698541 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:03.698547 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:03.699331 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:03.699336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:03.699959 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:03.699968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:03.700612 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:03.700617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:03.700766 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:03.704823 (+ 4057us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:42:03.704842 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:42:03.707116 (+ 2274us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39789: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":112,"threads_started":1}
W20260619 16:42:03.707620 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:42:03.697476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39271 (local address 127.0.98.62:39789)
0619 16:42:03.697713 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:03.697720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:03.697892 (+ 172us) server_negotiation.cc:407] Connection header received
0619 16:42:03.698055 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:03.698059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:03.698107 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:03.698206 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:42:03.698654 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:03.699180 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:03.700072 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:03.700391 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:03.700504 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:03.707460 (+ 6956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39271: BlockingRecv error: recv got EOF from 127.0.98.2:39271 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
I20260619 16:42:03.774755 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 9117
I20260619 16:42:03.780864 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 9269
I20260619 16:42:03.786638 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 9401
I20260619 16:42:03.792604 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 9047
2026-06-19T16:42:03Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18345 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T16:42:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:42:03Z Disabled control of system clock
I20260619 16:42:03.827124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35257
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 16:42:03.946852 9597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:42:03.947177 9597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:42:03.947216 9597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:42:03.951519 9597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:42:03.951611 9597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:42:03.951627 9597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:42:03.951648 9597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:42:03.951731 9597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:42:03.957259 9597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35257
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:42:03.958452 9597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:42:03.960096 9597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:42:03.970295 9602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:42:03.970328 9603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:42:03.970301 9605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:42:03.970412 9597 server_base.cc:1034] running on GCE node
I20260619 16:42:03.971388 9597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:42:03.973085 9597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:42:03.974316 9597 hybrid_clock.cc:648] HybridClock initialized: now 1781887323974275 us; error 69 us; skew 500 ppm
I20260619 16:42:03.976794 9597 webserver.cc:460] Webserver started at http://127.0.98.62:43381/ using document root <none> and password file <none>
I20260619 16:42:03.977510 9597 fs_manager.cc:359] Metadata directory not provided
I20260619 16:42:03.977586 9597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:42:03.977811 9597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:42:03.980134 9597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "d4e267f3eaee417eb93fb6f0179a8ef8"
format_stamp: "Formatted at 2026-06-19 16:42:03 on dist-test-slave-03cd"
server_key: "d0c675adda3200c0856605744907fddd"
server_key_iv: "0d4dad3f1db24b4a7d91ced038c8f134"
server_key_version: "encryptionkey@0"
I20260619 16:42:03.980775 9597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "d4e267f3eaee417eb93fb6f0179a8ef8"
format_stamp: "Formatted at 2026-06-19 16:42:03 on dist-test-slave-03cd"
server_key: "d0c675adda3200c0856605744907fddd"
server_key_iv: "0d4dad3f1db24b4a7d91ced038c8f134"
server_key_version: "encryptionkey@0"
I20260619 16:42:03.985997 9597 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:42:03.989360 9611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:42:03.991151 9597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:42:03.991390 9597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "d4e267f3eaee417eb93fb6f0179a8ef8"
format_stamp: "Formatted at 2026-06-19 16:42:03 on dist-test-slave-03cd"
server_key: "d0c675adda3200c0856605744907fddd"
server_key_iv: "0d4dad3f1db24b4a7d91ced038c8f134"
server_key_version: "encryptionkey@0"
I20260619 16:42:03.991545 9597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:42:04.004426 9597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:42:04.005193 9597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:42:04.005419 9597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:42:04.014041 9597 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35257
I20260619 16:42:04.014154 9663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35257 every 8 connection(s)
I20260619 16:42:04.015239 9597 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:42:04.019737 9664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:42:04.025043 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 9597
I20260619 16:42:04.025203 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:42:04.025563 392 external_mini_cluster.cc:1428] Setting key faec5f87f0182aeaaf4c2f5e632dd7f7
I20260619 16:42:04.028193 9664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4e267f3eaee417eb93fb6f0179a8ef8: Bootstrap starting.
I20260619 16:42:04.032132 9664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4e267f3eaee417eb93fb6f0179a8ef8: Neither blocks nor log segments found. Creating new log.
I20260619 16:42:04.033327 9664 log.cc:824] T 00000000000000000000000000000000 P d4e267f3eaee417eb93fb6f0179a8ef8: Log is configured to *not* fsync() on all Append() calls
I20260619 16:42:04.036700 9664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4e267f3eaee417eb93fb6f0179a8ef8: No bootstrap required, opened a new log
I20260619 16:42:04.039942 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:04.027493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37550 (local address 127.0.98.62:35257)
0619 16:42:04.028471 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:04.028497 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:04.028561 (+ 64us) server_negotiation.cc:407] Connection header received
0619 16:42:04.029629 (+ 1068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:04.029658 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:04.030598 (+ 940us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:04.031153 (+ 555us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:42:04.031899 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:04.033585 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:04.034954 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:04.035821 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:04.036045 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:04.037134 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:42:04.037142 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:42:04.037166 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:42:04.037798 (+ 632us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:42:04.037894 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:42:04.037927 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:42:04.038040 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:42:04.038273 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 16:42:04.038635 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":224,"threads_started":1}
I20260619 16:42:04.042295 9664 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4e267f3eaee417eb93fb6f0179a8ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e267f3eaee417eb93fb6f0179a8ef8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35257 } }
I20260619 16:42:04.042658 9664 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4e267f3eaee417eb93fb6f0179a8ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:42:04.042699 9664 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4e267f3eaee417eb93fb6f0179a8ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e267f3eaee417eb93fb6f0179a8ef8, State: Initialized, Role: FOLLOWER
I20260619 16:42:04.043367 9664 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4e267f3eaee417eb93fb6f0179a8ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e267f3eaee417eb93fb6f0179a8ef8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35257 } }
I20260619 16:42:04.043560 9664 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4e267f3eaee417eb93fb6f0179a8ef8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:42:04.043722 9664 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4e267f3eaee417eb93fb6f0179a8ef8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:42:04.043967 9664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4e267f3eaee417eb93fb6f0179a8ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:42:04.045601 9664 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4e267f3eaee417eb93fb6f0179a8ef8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e267f3eaee417eb93fb6f0179a8ef8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35257 } }
I20260619 16:42:04.046175 9664 leader_election.cc:302] T 00000000000000000000000000000000 P d4e267f3eaee417eb93fb6f0179a8ef8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4e267f3eaee417eb93fb6f0179a8ef8; no voters:
I20260619 16:42:04.046625 9664 leader_election.cc:288] T 00000000000000000000000000000000 P d4e267f3eaee417eb93fb6f0179a8ef8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:42:04.046921 9669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4e267f3eaee417eb93fb6f0179a8ef8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:42:04.047266 9669 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4e267f3eaee417eb93fb6f0179a8ef8 [term 1 LEADER]: Becoming Leader. State: Replica: d4e267f3eaee417eb93fb6f0179a8ef8, State: Running, Role: LEADER
I20260619 16:42:04.047739 9669 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4e267f3eaee417eb93fb6f0179a8ef8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e267f3eaee417eb93fb6f0179a8ef8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35257 } }
I20260619 16:42:04.048621 9664 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4e267f3eaee417eb93fb6f0179a8ef8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:42:04.049778 9670 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4e267f3eaee417eb93fb6f0179a8ef8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4e267f3eaee417eb93fb6f0179a8ef8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e267f3eaee417eb93fb6f0179a8ef8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35257 } } }
I20260619 16:42:04.049758 9671 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4e267f3eaee417eb93fb6f0179a8ef8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4e267f3eaee417eb93fb6f0179a8ef8. Latest consensus state: current_term: 1 leader_uuid: "d4e267f3eaee417eb93fb6f0179a8ef8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e267f3eaee417eb93fb6f0179a8ef8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35257 } } }
I20260619 16:42:04.049968 9670 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4e267f3eaee417eb93fb6f0179a8ef8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:42:04.049968 9671 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4e267f3eaee417eb93fb6f0179a8ef8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:42:04.050741 9678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:42:04.055445 9678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:42:04.063174 9678 catalog_manager.cc:1347] Generated new cluster ID: e1147751c7794ca8bd130e119450d1f6
I20260619 16:42:04.063337 9678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:42:04.080838 9678 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:42:04.082196 9678 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:42:04.088874 9678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4e267f3eaee417eb93fb6f0179a8ef8: Generated new TSK 0
I20260619 16:42:04.089766 9678 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:42:04.099993 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35257
--builtin_ntp_servers=127.0.98.20:32979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 16:42:04.225915 9688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:42:04.226245 9688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:42:04.226286 9688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:42:04.230675 9688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:42:04.230798 9688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:42:04.230881 9688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:42:04.235903 9688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:42:04.237190 9688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:42:04.238662 9688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:42:04.249204 9696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:42:04.249238 9693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:42:04.249204 9694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:42:04.249679 9688 server_base.cc:1034] running on GCE node
I20260619 16:42:04.250350 9688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:42:04.251294 9688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:42:04.252471 9688 hybrid_clock.cc:648] HybridClock initialized: now 1781887324252448 us; error 48 us; skew 500 ppm
I20260619 16:42:04.255013 9688 webserver.cc:460] Webserver started at http://127.0.98.1:42303/ using document root <none> and password file <none>
I20260619 16:42:04.255797 9688 fs_manager.cc:359] Metadata directory not provided
I20260619 16:42:04.255873 9688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:42:04.256124 9688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:42:04.258903 9688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/data/instance:
uuid: "e78cda1993bf4fddae84fef75a111407"
format_stamp: "Formatted at 2026-06-19 16:42:04 on dist-test-slave-03cd"
server_key: "9afb3ac4ffed8e2a77e9a50c2228949e"
server_key_iv: "75994f39b6ca4398857ccddf7d5c77de"
server_key_version: "encryptionkey@0"
I20260619 16:42:04.259768 9688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e78cda1993bf4fddae84fef75a111407"
format_stamp: "Formatted at 2026-06-19 16:42:04 on dist-test-slave-03cd"
server_key: "9afb3ac4ffed8e2a77e9a50c2228949e"
server_key_iv: "75994f39b6ca4398857ccddf7d5c77de"
server_key_version: "encryptionkey@0"
I20260619 16:42:04.264762 9688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 16:42:04.268262 9702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:42:04.270150 9688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:42:04.270455 9688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "e78cda1993bf4fddae84fef75a111407"
format_stamp: "Formatted at 2026-06-19 16:42:04 on dist-test-slave-03cd"
server_key: "9afb3ac4ffed8e2a77e9a50c2228949e"
server_key_iv: "75994f39b6ca4398857ccddf7d5c77de"
server_key_version: "encryptionkey@0"
I20260619 16:42:04.270622 9688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:42:04.307996 9688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:42:04.309044 9688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:42:04.309319 9688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:42:04.310158 9688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:42:04.311465 9688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:42:04.311545 9688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:42:04.311619 9688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:42:04.311659 9688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:42:04.324525 9688 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34439
I20260619 16:42:04.324739 9815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34439 every 8 connection(s)
I20260619 16:42:04.326118 9688 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:42:04.329128 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 9688
I20260619 16:42:04.329319 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:42:04.329670 392 external_mini_cluster.cc:1428] Setting key b0d110eed5c7a4005dc38f260802beb4
I20260619 16:42:04.332432 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35257
--builtin_ntp_servers=127.0.98.20:32979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 16:42:04.338946 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:04.328630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52799 (local address 127.0.98.62:35257)
0619 16:42:04.328878 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:04.328886 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:04.330113 (+ 1227us) server_negotiation.cc:407] Connection header received
0619 16:42:04.332006 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:04.332010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:04.332087 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:04.332203 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:42:04.333337 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:04.334239 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:04.336081 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:04.336606 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:04.336776 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:04.338137 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:42:04.338144 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:42:04.338160 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:42:04.338187 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:42:04.338227 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:42:04.338249 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:42:04.338361 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 16:42:04.338579 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 16:42:04.338682 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 16:42:04.339993 9818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:04.329052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35257 (local address 127.0.98.1:52799)
0619 16:42:04.329912 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0619 16:42:04.329970 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 16:42:04.331770 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:42:04.332369 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:42:04.332377 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:42:04.332510 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:42:04.333162 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:04.333177 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:04.334412 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:04.334416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:04.335916 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:04.335930 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:04.336808 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:04.336811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:04.337020 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:04.337794 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:42:04.337796 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:42:04.337844 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:42:04.337930 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:42:04.338394 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:42:04.338402 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:42:04.338408 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:42:04.338552 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 16:42:04.338837 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":198,"threads_started":1}
I20260619 16:42:04.341607 9816 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35257
I20260619 16:42:04.341892 9816 heartbeater.cc:464] Registering TS with master...
I20260619 16:42:04.342521 9816 heartbeater.cc:511] Master 127.0.98.62:35257 requested a full tablet report, sending...
I20260619 16:42:04.344726 9627 ts_manager.cc:194] Registered new tserver with Master: e78cda1993bf4fddae84fef75a111407 (127.0.98.1:34439)
I20260619 16:42:04.346661 9627 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:52799
W20260619 16:42:04.465631 9819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:42:04.465999 9819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:42:04.466041 9819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:42:04.470434 9819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:42:04.470520 9819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:42:04.470585 9819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:42:04.476207 9819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:42:04.477659 9819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:42:04.479352 9819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:42:04.491806 9824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:42:04.491750 9827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:42:04.491812 9825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:42:04.492369 9819 server_base.cc:1034] running on GCE node
I20260619 16:42:04.492897 9819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:42:04.493687 9819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:42:04.494926 9819 hybrid_clock.cc:648] HybridClock initialized: now 1781887324494882 us; error 63 us; skew 500 ppm
I20260619 16:42:04.497320 9819 webserver.cc:460] Webserver started at http://127.0.98.2:36429/ using document root <none> and password file <none>
I20260619 16:42:04.498004 9819 fs_manager.cc:359] Metadata directory not provided
I20260619 16:42:04.498078 9819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:42:04.498314 9819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:42:04.500797 9819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/data/instance:
uuid: "3b7a0d4cf900457486ca2879237d5c08"
format_stamp: "Formatted at 2026-06-19 16:42:04 on dist-test-slave-03cd"
server_key: "1e46578cdee266c09ee3a780e0dfd46a"
server_key_iv: "eca284e3ea8b72a7116729f1d0b7abfb"
server_key_version: "encryptionkey@0"
I20260619 16:42:04.501432 9819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3b7a0d4cf900457486ca2879237d5c08"
format_stamp: "Formatted at 2026-06-19 16:42:04 on dist-test-slave-03cd"
server_key: "1e46578cdee266c09ee3a780e0dfd46a"
server_key_iv: "eca284e3ea8b72a7116729f1d0b7abfb"
server_key_version: "encryptionkey@0"
I20260619 16:42:04.506085 9819 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:42:04.509668 9833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:42:04.511445 9819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260619 16:42:04.511633 9819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "3b7a0d4cf900457486ca2879237d5c08"
format_stamp: "Formatted at 2026-06-19 16:42:04 on dist-test-slave-03cd"
server_key: "1e46578cdee266c09ee3a780e0dfd46a"
server_key_iv: "eca284e3ea8b72a7116729f1d0b7abfb"
server_key_version: "encryptionkey@0"
I20260619 16:42:04.511844 9819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:42:04.554508 9819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:42:04.555294 9819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:42:04.555512 9819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:42:04.556281 9819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:42:04.557382 9819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:42:04.557464 9819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:42:04.557540 9819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:42:04.557588 9819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:42:04.567541 9819 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46743
I20260619 16:42:04.567559 9946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46743 every 8 connection(s)
I20260619 16:42:04.568598 9819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:42:04.572252 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 9819
I20260619 16:42:04.572369 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:42:04.572678 392 external_mini_cluster.cc:1428] Setting key 346c7da6f4c84ceab4c98daacaf5fe40
I20260619 16:42:04.575222 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35257
--builtin_ntp_servers=127.0.98.20:32979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 16:42:04.580914 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:04.571070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49967 (local address 127.0.98.62:35257)
0619 16:42:04.571250 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:04.571257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:04.572682 (+ 1425us) server_negotiation.cc:407] Connection header received
0619 16:42:04.574582 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:04.574585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:04.574649 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:04.574738 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:42:04.576159 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:04.576868 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:04.578244 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:04.578625 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:04.578750 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:04.580357 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:42:04.580361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:42:04.580363 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:42:04.580379 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:42:04.580405 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:42:04.580414 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:42:04.580483 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 16:42:04.580693 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 16:42:04.580765 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:42:04.582268 9949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:04.571473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35257 (local address 127.0.98.2:49967)
0619 16:42:04.572468 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0619 16:42:04.572535 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 16:42:04.574322 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:42:04.574974 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:42:04.574982 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:42:04.575142 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:42:04.575985 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:04.576002 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:04.577001 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:04.577005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:04.578108 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:04.578119 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:04.578865 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:04.578871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:04.579120 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:04.579942 (+ 822us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:42:04.579945 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:42:04.580009 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:42:04.580137 (+ 128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:42:04.580510 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:42:04.580522 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:42:04.580531 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 16:42:04.580676 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 16:42:04.581037 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":194,"threads_started":1}
I20260619 16:42:04.583925 9947 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35257
I20260619 16:42:04.584268 9947 heartbeater.cc:464] Registering TS with master...
I20260619 16:42:04.584986 9947 heartbeater.cc:511] Master 127.0.98.62:35257 requested a full tablet report, sending...
I20260619 16:42:04.586467 9627 ts_manager.cc:194] Registered new tserver with Master: 3b7a0d4cf900457486ca2879237d5c08 (127.0.98.2:46743)
I20260619 16:42:04.587476 9627 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:49967
W20260619 16:42:04.698875 9950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:42:04.699168 9950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:42:04.699208 9950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:42:04.704772 9950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:42:04.704852 9950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:42:04.704914 9950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:42:04.710283 9950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:42:04.711860 9950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:42:04.713407 9950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:42:04.724699 9956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:42:04.724710 9958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:42:04.724800 9950 server_base.cc:1034] running on GCE node
W20260619 16:42:04.725036 9955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:42:04.725459 9950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:42:04.726123 9950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:42:04.727351 9950 hybrid_clock.cc:648] HybridClock initialized: now 1781887324727332 us; error 31 us; skew 500 ppm
I20260619 16:42:04.729465 9950 webserver.cc:460] Webserver started at http://127.0.98.3:46515/ using document root <none> and password file <none>
I20260619 16:42:04.730067 9950 fs_manager.cc:359] Metadata directory not provided
I20260619 16:42:04.730165 9950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:42:04.730396 9950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:42:04.732635 9950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/data/instance:
uuid: "0da81dd0cc4a4d2d8f922f507ebc948f"
format_stamp: "Formatted at 2026-06-19 16:42:04 on dist-test-slave-03cd"
server_key: "70b3c26a3a9451b57e5379fef9e28ed8"
server_key_iv: "7e2721976fecfaba06c18c13f33a671f"
server_key_version: "encryptionkey@0"
I20260619 16:42:04.733218 9950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0da81dd0cc4a4d2d8f922f507ebc948f"
format_stamp: "Formatted at 2026-06-19 16:42:04 on dist-test-slave-03cd"
server_key: "70b3c26a3a9451b57e5379fef9e28ed8"
server_key_iv: "7e2721976fecfaba06c18c13f33a671f"
server_key_version: "encryptionkey@0"
I20260619 16:42:04.737011 9950 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:42:04.740031 9964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:42:04.741389 9950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:42:04.741540 9950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "0da81dd0cc4a4d2d8f922f507ebc948f"
format_stamp: "Formatted at 2026-06-19 16:42:04 on dist-test-slave-03cd"
server_key: "70b3c26a3a9451b57e5379fef9e28ed8"
server_key_iv: "7e2721976fecfaba06c18c13f33a671f"
server_key_version: "encryptionkey@0"
I20260619 16:42:04.741670 9950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:42:04.764376 9950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:42:04.765208 9950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:42:04.765466 9950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:42:04.766165 9950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:42:04.767320 9950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:42:04.767402 9950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:42:04.767498 9950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:42:04.767547 9950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:42:04.778543 9950 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44789
I20260619 16:42:04.778579 10077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44789 every 8 connection(s)
I20260619 16:42:04.779598 9950 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:42:04.785475 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 9950
I20260619 16:42:04.785580 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:42:04.785884 392 external_mini_cluster.cc:1428] Setting key 5a99e84010be7b9f547953d4d3c8a4f2
I20260619 16:42:04.789839 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:04.781662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48723 (local address 127.0.98.62:35257)
0619 16:42:04.781837 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:04.781842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:04.782958 (+ 1116us) server_negotiation.cc:407] Connection header received
0619 16:42:04.784631 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:04.784634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:04.784687 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:04.784777 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:42:04.785924 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:04.786558 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:04.787837 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:04.788066 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:04.788181 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:04.789332 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:42:04.789336 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:42:04.789338 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:42:04.789352 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:42:04.789370 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:42:04.789378 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:42:04.789452 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:42:04.789617 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 16:42:04.789683 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:42:04.790805 10080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:04.782036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35257 (local address 127.0.98.3:48723)
0619 16:42:04.782794 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0619 16:42:04.782841 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:42:04.784399 (+ 1558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:42:04.784949 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:42:04.784956 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:42:04.785051 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:42:04.785767 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:04.785786 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:04.786708 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:04.786717 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:04.787646 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:04.787723 (+ 77us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:04.788227 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:04.788230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:04.788417 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:04.789050 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:42:04.789051 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:42:04.789094 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:42:04.789169 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:42:04.789464 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:42:04.789476 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:42:04.789482 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:42:04.789617 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 16:42:04.789936 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":114,"threads_started":1}
I20260619 16:42:04.792075 10078 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35257
I20260619 16:42:04.792333 10078 heartbeater.cc:464] Registering TS with master...
I20260619 16:42:04.792896 10078 heartbeater.cc:511] Master 127.0.98.62:35257 requested a full tablet report, sending...
I20260619 16:42:04.793999 9627 ts_manager.cc:194] Registered new tserver with Master: 0da81dd0cc4a4d2d8f922f507ebc948f (127.0.98.3:44789)
I20260619 16:42:04.794603 9627 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:48723
I20260619 16:42:04.800108 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 16:42:04.802888 9667 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:42:04.802067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46594 (local address 127.0.98.62:35257)
0619 16:42:04.802186 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:04.802190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:04.802334 (+ 144us) server_negotiation.cc:407] Connection header received
0619 16:42:04.802434 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:04.802436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:04.802475 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:04.802553 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:42:04.802772 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46594: BlockingRecv error: recv got EOF from 127.0.0.1:46594 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 16:42:04.803098 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 9688
I20260619 16:42:04.808578 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 9819
I20260619 16:42:04.813923 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 9950
I20260619 16:42:04.819252 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 9597
2026-06-19T16:42:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (1024 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T16:42:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:42:04Z Disabled control of system clock
I20260619 16:42:04.849313 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44197
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 16:42:04.969159 10089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:42:04.969489 10089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:42:04.969529 10089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:42:04.973752 10089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:42:04.973835 10089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:42:04.973850 10089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:42:04.973868 10089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:42:04.973884 10089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:42:04.979100 10089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44197
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:42:04.980360 10089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:42:04.981794 10089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:42:04.990669 10095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:42:04.991079 10094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:42:04.991096 10097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:42:04.991837 10089 server_base.cc:1034] running on GCE node
I20260619 16:42:04.992496 10089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:42:04.993805 10089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:42:04.994995 10089 hybrid_clock.cc:648] HybridClock initialized: now 1781887324994981 us; error 40 us; skew 500 ppm
I20260619 16:42:04.997208 10089 webserver.cc:460] Webserver started at http://127.0.98.62:37209/ using document root <none> and password file <none>
I20260619 16:42:04.997840 10089 fs_manager.cc:359] Metadata directory not provided
I20260619 16:42:04.997936 10089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:42:04.998178 10089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:42:05.000473 10089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "ffcd30766e1a476a8ae9e47786600b76"
format_stamp: "Formatted at 2026-06-19 16:42:04 on dist-test-slave-03cd"
server_key: "f88a950ba11fe9dd68aca9ca77707296"
server_key_iv: "3f7eec6aa85c5d5907b34796379310fe"
server_key_version: "encryptionkey@0"
I20260619 16:42:05.001087 10089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "ffcd30766e1a476a8ae9e47786600b76"
format_stamp: "Formatted at 2026-06-19 16:42:04 on dist-test-slave-03cd"
server_key: "f88a950ba11fe9dd68aca9ca77707296"
server_key_iv: "3f7eec6aa85c5d5907b34796379310fe"
server_key_version: "encryptionkey@0"
I20260619 16:42:05.005599 10089 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 16:42:05.008826 10103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:42:05.010291 10089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:42:05.010471 10089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "ffcd30766e1a476a8ae9e47786600b76"
format_stamp: "Formatted at 2026-06-19 16:42:04 on dist-test-slave-03cd"
server_key: "f88a950ba11fe9dd68aca9ca77707296"
server_key_iv: "3f7eec6aa85c5d5907b34796379310fe"
server_key_version: "encryptionkey@0"
I20260619 16:42:05.010610 10089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:42:05.016521 10089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:42:05.017266 10089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:42:05.017475 10089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:42:05.025916 10089 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44197
I20260619 16:42:05.025964 10155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44197 every 8 connection(s)
I20260619 16:42:05.027237 10089 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:42:05.030646 10156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:42:05.036055 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 10089
I20260619 16:42:05.036176 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:42:05.036460 392 external_mini_cluster.cc:1428] Setting key d2a0bf218b35c3f7428683e05d5a58bc
I20260619 16:42:05.036863 10156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffcd30766e1a476a8ae9e47786600b76: Bootstrap starting.
I20260619 16:42:05.039748 10156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffcd30766e1a476a8ae9e47786600b76: Neither blocks nor log segments found. Creating new log.
I20260619 16:42:05.040728 10156 log.cc:824] T 00000000000000000000000000000000 P ffcd30766e1a476a8ae9e47786600b76: Log is configured to *not* fsync() on all Append() calls
I20260619 16:42:05.042762 10159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:05.037914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37666 (local address 127.0.98.62:44197)
0619 16:42:05.038640 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:05.038656 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:05.038702 (+ 46us) server_negotiation.cc:407] Connection header received
0619 16:42:05.039396 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:05.039414 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:05.039925 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:05.040229 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:42:05.041156 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:42:05.041162 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:42:05.041174 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:42:05.041185 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:42:05.041247 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:42:05.041285 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:42:05.041388 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 16:42:05.041536 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 16:42:05.041825 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":174,"threads_started":1}
I20260619 16:42:05.043391 10156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffcd30766e1a476a8ae9e47786600b76: No bootstrap required, opened a new log
I20260619 16:42:05.047360 10156 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffcd30766e1a476a8ae9e47786600b76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffcd30766e1a476a8ae9e47786600b76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44197 } }
I20260619 16:42:05.047646 10156 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffcd30766e1a476a8ae9e47786600b76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:42:05.047770 10156 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffcd30766e1a476a8ae9e47786600b76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffcd30766e1a476a8ae9e47786600b76, State: Initialized, Role: FOLLOWER
I20260619 16:42:05.048197 10156 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffcd30766e1a476a8ae9e47786600b76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffcd30766e1a476a8ae9e47786600b76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44197 } }
I20260619 16:42:05.048319 10156 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffcd30766e1a476a8ae9e47786600b76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:42:05.048362 10156 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffcd30766e1a476a8ae9e47786600b76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:42:05.048486 10156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffcd30766e1a476a8ae9e47786600b76 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:42:05.049654 10156 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffcd30766e1a476a8ae9e47786600b76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffcd30766e1a476a8ae9e47786600b76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44197 } }
I20260619 16:42:05.050000 10156 leader_election.cc:302] T 00000000000000000000000000000000 P ffcd30766e1a476a8ae9e47786600b76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffcd30766e1a476a8ae9e47786600b76; no voters:
I20260619 16:42:05.050323 10156 leader_election.cc:288] T 00000000000000000000000000000000 P ffcd30766e1a476a8ae9e47786600b76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:42:05.050506 10161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffcd30766e1a476a8ae9e47786600b76 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:42:05.050805 10161 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffcd30766e1a476a8ae9e47786600b76 [term 1 LEADER]: Becoming Leader. State: Replica: ffcd30766e1a476a8ae9e47786600b76, State: Running, Role: LEADER
I20260619 16:42:05.051481 10161 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffcd30766e1a476a8ae9e47786600b76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffcd30766e1a476a8ae9e47786600b76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44197 } }
I20260619 16:42:05.052011 10156 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffcd30766e1a476a8ae9e47786600b76 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:42:05.053387 10163 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffcd30766e1a476a8ae9e47786600b76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffcd30766e1a476a8ae9e47786600b76. Latest consensus state: current_term: 1 leader_uuid: "ffcd30766e1a476a8ae9e47786600b76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffcd30766e1a476a8ae9e47786600b76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44197 } } }
I20260619 16:42:05.053551 10163 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffcd30766e1a476a8ae9e47786600b76 [sys.catalog]: This master's current role is: LEADER
I20260619 16:42:05.054029 10170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:42:05.054029 10162 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffcd30766e1a476a8ae9e47786600b76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffcd30766e1a476a8ae9e47786600b76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffcd30766e1a476a8ae9e47786600b76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44197 } } }
I20260619 16:42:05.054171 10162 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffcd30766e1a476a8ae9e47786600b76 [sys.catalog]: This master's current role is: LEADER
I20260619 16:42:05.057480 10170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:42:05.063819 10170 catalog_manager.cc:1347] Generated new cluster ID: 488ef14c40ea46cba905ace9614f8a99
I20260619 16:42:05.063925 10170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:42:05.072496 10170 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:42:05.072898 10170 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:42:05.078310 10170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffcd30766e1a476a8ae9e47786600b76: Generated new TSK 0
I20260619 16:42:05.079344 10170 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:42:05.089979 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44197
--builtin_ntp_servers=127.0.98.20:33159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 16:42:05.211978 10180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:42:05.212275 10180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:42:05.212321 10180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:42:05.216714 10180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:42:05.216791 10180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:42:05.216854 10180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:42:05.221750 10180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:42:05.222863 10180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:42:05.224331 10180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:42:05.236085 10185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:42:05.236157 10180 server_base.cc:1034] running on GCE node
W20260619 16:42:05.236085 10186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:42:05.236460 10188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:42:05.236948 10180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:42:05.237674 10180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:42:05.238847 10180 hybrid_clock.cc:648] HybridClock initialized: now 1781887325238826 us; error 35 us; skew 500 ppm
I20260619 16:42:05.240823 10180 webserver.cc:460] Webserver started at http://127.0.98.1:35321/ using document root <none> and password file <none>
I20260619 16:42:05.241446 10180 fs_manager.cc:359] Metadata directory not provided
I20260619 16:42:05.241537 10180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:42:05.241762 10180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:42:05.243973 10180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/data/instance:
uuid: "c67bb349c0364bb7a7c064029db3c504"
format_stamp: "Formatted at 2026-06-19 16:42:05 on dist-test-slave-03cd"
server_key: "4a404f0a31510b77392cdb9fb154eb16"
server_key_iv: "78eea4d7a8119441e6878306d5ed3c0b"
server_key_version: "encryptionkey@0"
I20260619 16:42:05.244540 10180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c67bb349c0364bb7a7c064029db3c504"
format_stamp: "Formatted at 2026-06-19 16:42:05 on dist-test-slave-03cd"
server_key: "4a404f0a31510b77392cdb9fb154eb16"
server_key_iv: "78eea4d7a8119441e6878306d5ed3c0b"
server_key_version: "encryptionkey@0"
I20260619 16:42:05.248730 10180 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:42:05.251340 10194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:42:05.252666 10180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 16:42:05.252796 10180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "c67bb349c0364bb7a7c064029db3c504"
format_stamp: "Formatted at 2026-06-19 16:42:05 on dist-test-slave-03cd"
server_key: "4a404f0a31510b77392cdb9fb154eb16"
server_key_iv: "78eea4d7a8119441e6878306d5ed3c0b"
server_key_version: "encryptionkey@0"
I20260619 16:42:05.252928 10180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:42:05.257675 10180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:42:05.258318 10180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:42:05.258504 10180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:42:05.259105 10180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:42:05.260442 10180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:42:05.260511 10180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:42:05.260567 10180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:42:05.260607 10180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:42:05.271260 10180 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44377
I20260619 16:42:05.271282 10307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44377 every 8 connection(s)
I20260619 16:42:05.272260 10180 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:42:05.277580 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 10180
I20260619 16:42:05.277704 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:42:05.278084 392 external_mini_cluster.cc:1428] Setting key 606a65201b7b215d1306f1b59b7ec13c
I20260619 16:42:05.279019 10159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:05.274212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49889 (local address 127.0.98.62:44197)
0619 16:42:05.274376 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:05.274381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:05.275467 (+ 1086us) server_negotiation.cc:407] Connection header received
0619 16:42:05.276889 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:05.276893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:05.276934 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:05.277014 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:42:05.278424 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:42:05.278428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:42:05.278430 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:42:05.278443 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:42:05.278480 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:42:05.278493 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:42:05.278600 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 16:42:05.278787 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 16:42:05.278875 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:42:05.280114 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44197
--builtin_ntp_servers=127.0.98.20:33159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 16:42:05.280150 10310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:05.274594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44197 (local address 127.0.98.1:49889)
0619 16:42:05.275256 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0619 16:42:05.275342 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0619 16:42:05.276685 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:42:05.277170 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:42:05.277176 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:42:05.277315 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:42:05.278111 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:42:05.278114 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:42:05.278170 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:42:05.278258 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:42:05.278601 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:42:05.278609 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:42:05.278614 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:42:05.278817 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 16:42:05.279074 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":120,"threads_started":1}
I20260619 16:42:05.281337 10308 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44197
I20260619 16:42:05.281627 10308 heartbeater.cc:464] Registering TS with master...
I20260619 16:42:05.282140 10308 heartbeater.cc:511] Master 127.0.98.62:44197 requested a full tablet report, sending...
I20260619 16:42:05.283787 10120 ts_manager.cc:194] Registered new tserver with Master: c67bb349c0364bb7a7c064029db3c504 (127.0.98.1:44377)
W20260619 16:42:05.398272 10311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:42:05.398571 10311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:42:05.398612 10311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:42:05.403138 10311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:42:05.403237 10311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:42:05.403302 10311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:42:05.408713 10311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:42:05.409919 10311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:42:05.411375 10311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:42:05.422254 10311 server_base.cc:1034] running on GCE node
W20260619 16:42:05.422236 10317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:42:05.422474 10316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:42:05.422237 10319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:42:05.423084 10311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:42:05.423863 10311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:42:05.425069 10311 hybrid_clock.cc:648] HybridClock initialized: now 1781887325425051 us; error 34 us; skew 500 ppm
I20260619 16:42:05.427309 10311 webserver.cc:460] Webserver started at http://127.0.98.2:40757/ using document root <none> and password file <none>
I20260619 16:42:05.428038 10311 fs_manager.cc:359] Metadata directory not provided
I20260619 16:42:05.428130 10311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:42:05.428368 10311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:42:05.430562 10311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/data/instance:
uuid: "3d780041d3f44cd28c8ab60a04b7e543"
format_stamp: "Formatted at 2026-06-19 16:42:05 on dist-test-slave-03cd"
server_key: "8615252bc929df071f85ef9b7d2061d6"
server_key_iv: "d507a561fcbcbcab3dfcf0744d0128e8"
server_key_version: "encryptionkey@0"
I20260619 16:42:05.431178 10311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3d780041d3f44cd28c8ab60a04b7e543"
format_stamp: "Formatted at 2026-06-19 16:42:05 on dist-test-slave-03cd"
server_key: "8615252bc929df071f85ef9b7d2061d6"
server_key_iv: "d507a561fcbcbcab3dfcf0744d0128e8"
server_key_version: "encryptionkey@0"
I20260619 16:42:05.435246 10311 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260619 16:42:05.438112 10325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:42:05.439769 10311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:42:05.439920 10311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "3d780041d3f44cd28c8ab60a04b7e543"
format_stamp: "Formatted at 2026-06-19 16:42:05 on dist-test-slave-03cd"
server_key: "8615252bc929df071f85ef9b7d2061d6"
server_key_iv: "d507a561fcbcbcab3dfcf0744d0128e8"
server_key_version: "encryptionkey@0"
I20260619 16:42:05.440066 10311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:42:05.444868 10311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:42:05.445525 10311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:42:05.445724 10311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:42:05.446300 10311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:42:05.447273 10311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:42:05.447337 10311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:42:05.447394 10311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:42:05.447440 10311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:42:05.456693 10311 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38167
I20260619 16:42:05.456712 10438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38167 every 8 connection(s)
I20260619 16:42:05.457710 10311 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:42:05.464406 10159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:05.459521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50855 (local address 127.0.98.62:44197)
0619 16:42:05.459693 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:05.459698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:05.460837 (+ 1139us) server_negotiation.cc:407] Connection header received
0619 16:42:05.462157 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:05.462161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:05.462201 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:05.462289 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:42:05.463641 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:42:05.463647 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:42:05.463650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:42:05.463962 (+ 312us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:42:05.463983 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:42:05.463996 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:42:05.464088 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 16:42:05.464243 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 16:42:05.464284 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:42:05.465415 10441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:05.459979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44197 (local address 127.0.98.2:50855)
0619 16:42:05.460689 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0619 16:42:05.460733 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:42:05.461934 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:42:05.462473 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:42:05.462480 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:42:05.462575 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:42:05.463191 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:42:05.463193 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:42:05.463235 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:42:05.463311 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:42:05.464090 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:42:05.464098 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:42:05.464103 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:42:05.464297 (+ 194us) client_negotiation.cc:240] Negotiation successful
0619 16:42:05.464556 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":116,"threads_started":1}
I20260619 16:42:05.466862 10439 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44197
I20260619 16:42:05.467145 10439 heartbeater.cc:464] Registering TS with master...
I20260619 16:42:05.467620 10439 heartbeater.cc:511] Master 127.0.98.62:44197 requested a full tablet report, sending...
I20260619 16:42:05.467634 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 10311
I20260619 16:42:05.467831 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:42:05.468129 392 external_mini_cluster.cc:1428] Setting key ac3f0f01e303f52d35afc5b1570a4bfc
I20260619 16:42:05.468967 10120 ts_manager.cc:194] Registered new tserver with Master: 3d780041d3f44cd28c8ab60a04b7e543 (127.0.98.2:38167)
I20260619 16:42:05.470281 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44197
--builtin_ntp_servers=127.0.98.20:33159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 16:42:05.586362 10442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:42:05.586655 10442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:42:05.586697 10442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:42:05.591178 10442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:42:05.591259 10442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:42:05.591351 10442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:42:05.596429 10442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:42:05.597620 10442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:42:05.599076 10442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:42:05.609146 10447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:42:05.609163 10450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:42:05.609241 10442 server_base.cc:1034] running on GCE node
W20260619 16:42:05.609169 10448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:42:05.609825 10442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:42:05.610539 10442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:42:05.611764 10442 hybrid_clock.cc:648] HybridClock initialized: now 1781887325611707 us; error 67 us; skew 500 ppm
I20260619 16:42:05.613871 10442 webserver.cc:460] Webserver started at http://127.0.98.3:33563/ using document root <none> and password file <none>
I20260619 16:42:05.614513 10442 fs_manager.cc:359] Metadata directory not provided
I20260619 16:42:05.614605 10442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:42:05.614858 10442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:42:05.617447 10442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/data/instance:
uuid: "c5eb9e5bc3074bedad2d48811cbec9f5"
format_stamp: "Formatted at 2026-06-19 16:42:05 on dist-test-slave-03cd"
server_key: "959a7df0221ad9b1ae922aeddf2514c7"
server_key_iv: "751d8380f6ccf796b3a5dd9f6b3da31f"
server_key_version: "encryptionkey@0"
I20260619 16:42:05.618039 10442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c5eb9e5bc3074bedad2d48811cbec9f5"
format_stamp: "Formatted at 2026-06-19 16:42:05 on dist-test-slave-03cd"
server_key: "959a7df0221ad9b1ae922aeddf2514c7"
server_key_iv: "751d8380f6ccf796b3a5dd9f6b3da31f"
server_key_version: "encryptionkey@0"
I20260619 16:42:05.622004 10442 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:42:05.624753 10456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:42:05.626238 10442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:42:05.626415 10442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "c5eb9e5bc3074bedad2d48811cbec9f5"
format_stamp: "Formatted at 2026-06-19 16:42:05 on dist-test-slave-03cd"
server_key: "959a7df0221ad9b1ae922aeddf2514c7"
server_key_iv: "751d8380f6ccf796b3a5dd9f6b3da31f"
server_key_version: "encryptionkey@0"
I20260619 16:42:05.626549 10442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:42:05.632388 10442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:42:05.633292 10442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:42:05.633522 10442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:42:05.634222 10442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:42:05.635399 10442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:42:05.635471 10442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:42:05.635550 10442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:42:05.635591 10442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:42:05.647220 10442 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36487
I20260619 16:42:05.647281 10569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36487 every 8 connection(s)
I20260619 16:42:05.648545 10442 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:42:05.656327 10159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:05.650722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40357 (local address 127.0.98.62:44197)
0619 16:42:05.650959 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:05.650963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:05.652302 (+ 1339us) server_negotiation.cc:407] Connection header received
0619 16:42:05.654024 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:05.654028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:05.654119 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:05.654208 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:42:05.655616 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:42:05.655620 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:42:05.655622 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:42:05.655635 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:42:05.655812 (+ 177us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:42:05.655826 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:42:05.655933 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 16:42:05.656125 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 16:42:05.656175 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260619 16:42:05.657285 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 10442
I20260619 16:42:05.657403 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887259574980-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:42:05.657605 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:05.651108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44197 (local address 127.0.98.3:40357)
0619 16:42:05.652077 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0619 16:42:05.652132 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 16:42:05.653734 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:42:05.654459 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:42:05.654470 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:42:05.654582 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:42:05.655286 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:42:05.655288 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:42:05.655335 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:42:05.655415 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:42:05.655934 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:42:05.655942 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:42:05.655947 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:42:05.656164 (+ 217us) client_negotiation.cc:240] Negotiation successful
0619 16:42:05.656413 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":150,"threads_started":1}
I20260619 16:42:05.657776 392 external_mini_cluster.cc:1428] Setting key bfb057da0830f39b84b800c7f50f3eed
I20260619 16:42:05.659217 10570 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44197
I20260619 16:42:05.659528 10570 heartbeater.cc:464] Registering TS with master...
I20260619 16:42:05.660072 10570 heartbeater.cc:511] Master 127.0.98.62:44197 requested a full tablet report, sending...
I20260619 16:42:05.661289 10120 ts_manager.cc:194] Registered new tserver with Master: c5eb9e5bc3074bedad2d48811cbec9f5 (127.0.98.3:36487)
I20260619 16:42:05.672845 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 16:42:05.676498 10159 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:42:05.675235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37680 (local address 127.0.98.62:44197)
0619 16:42:05.675426 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:05.675432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:05.675636 (+ 204us) server_negotiation.cc:407] Connection header received
0619 16:42:05.675889 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:05.675892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:05.675939 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:05.676028 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:42:05.676313 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37680: BlockingRecv error: recv got EOF from 127.0.0.1:37680 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:42:05.676741 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 10180
I20260619 16:42:05.683022 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 10311
I20260619 16:42:05.688858 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 10442
I20260619 16:42:05.694640 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 10089
2026-06-19T16:42:05Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (875 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:42:05 dist-test-slave-03cd krb5kdc[10580](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:42:05 dist-test-slave-03cd krb5kdc[10580](info): set up 2 sockets
Jun 19 16:42:05 dist-test-slave-03cd krb5kdc[10580](info): commencing operation
krb5kdc: starting...
W20260619 16:42:06.753274 392 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:42:06 dist-test-slave-03cd krb5kdc[10580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:42:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:42:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:42:06.904126 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34357
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:42:07.019862 10593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:42:07.020229 10593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:42:07.020305 10593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:42:07.024624 10593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:42:07.024746 10593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:42:07.024782 10593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:42:07.024820 10593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:42:07.024852 10593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:42:07.030820 10593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:42:07.032120 10593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:42:07.033236 10593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:42:07.042497 10601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:42:07.042493 10599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:42:07.042521 10598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:42:07.043081 10593 server_base.cc:1034] running on GCE node
I20260619 16:42:07.044080 10593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:42:07.045651 10593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:42:07.046947 10593 hybrid_clock.cc:648] HybridClock initialized: now 1781887327046935 us; error 108 us; skew 500 ppm
Jun 19 16:42:07 dist-test-slave-03cd krb5kdc[10580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:42:07.050797 10593 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:42:07.052132 10593 webserver.cc:460] Webserver started at http://127.0.98.62:42049/ using document root <none> and password file <none>
I20260619 16:42:07.052809 10593 fs_manager.cc:359] Metadata directory not provided
I20260619 16:42:07.052877 10593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:42:07.053097 10593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:42:07.055652 10593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "a4aaeb87b0114f52a4a0b03e183d344d"
format_stamp: "Formatted at 2026-06-19 16:42:07 on dist-test-slave-03cd"
server_key: "c1349cf79a9f70bbeb3f0451988b8c71"
server_key_iv: "8e14f24b59ade6a26961dc695aa01d44"
server_key_version: "encryptionkey@0"
I20260619 16:42:07.056407 10593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "a4aaeb87b0114f52a4a0b03e183d344d"
format_stamp: "Formatted at 2026-06-19 16:42:07 on dist-test-slave-03cd"
server_key: "c1349cf79a9f70bbeb3f0451988b8c71"
server_key_iv: "8e14f24b59ade6a26961dc695aa01d44"
server_key_version: "encryptionkey@0"
I20260619 16:42:07.061347 10593 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 16:42:07.065057 10608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:42:07.066994 10593 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:42:07.067303 10593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "a4aaeb87b0114f52a4a0b03e183d344d"
format_stamp: "Formatted at 2026-06-19 16:42:07 on dist-test-slave-03cd"
server_key: "c1349cf79a9f70bbeb3f0451988b8c71"
server_key_iv: "8e14f24b59ade6a26961dc695aa01d44"
server_key_version: "encryptionkey@0"
I20260619 16:42:07.067493 10593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:42:07.086822 10593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:42:07.091202 10593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:42:07.091418 10593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:42:07.100474 10593 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34357
I20260619 16:42:07.100831 10660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34357 every 8 connection(s)
I20260619 16:42:07.101735 10593 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:42:07.102636 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 10593
I20260619 16:42:07.102768 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:42:07.103111 392 external_mini_cluster.cc:1428] Setting key eb1eb6ddb0b55a91c1152e7bb2a1a65b
I20260619 16:42:07.105754 10661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:42:07.113493 10661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4aaeb87b0114f52a4a0b03e183d344d: Bootstrap starting.
Jun 19 16:42:07 dist-test-slave-03cd krb5kdc[10580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:42:07.116439 10661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4aaeb87b0114f52a4a0b03e183d344d: Neither blocks nor log segments found. Creating new log.
I20260619 16:42:07.117334 10661 log.cc:824] T 00000000000000000000000000000000 P a4aaeb87b0114f52a4a0b03e183d344d: Log is configured to *not* fsync() on all Append() calls
I20260619 16:42:07.120015 10661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4aaeb87b0114f52a4a0b03e183d344d: No bootstrap required, opened a new log
I20260619 16:42:07.121392 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:07.104857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36054 (local address 127.0.98.62:34357)
0619 16:42:07.105336 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:07.105354 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:07.105401 (+ 47us) server_negotiation.cc:407] Connection header received
0619 16:42:07.106247 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:07.106270 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:07.106700 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:07.107077 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:42:07.108185 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:07.109351 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:07.110687 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:07.111363 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:07.111525 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:07.115205 (+ 3680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:42:07.115213 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:42:07.115235 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:42:07.115284 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:42:07.117682 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:42:07.118301 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:42:07.118308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:42:07.118314 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:42:07.118467 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:42:07.118784 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:42:07.118788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:42:07.118790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:42:07.119345 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:42:07.119597 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 16:42:07.119938 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 16:42:07.120384 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":142,"threads_started":1}
I20260619 16:42:07.124413 10661 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4aaeb87b0114f52a4a0b03e183d344d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4aaeb87b0114f52a4a0b03e183d344d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34357 } }
I20260619 16:42:07.124778 10661 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4aaeb87b0114f52a4a0b03e183d344d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:42:07.124823 10661 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4aaeb87b0114f52a4a0b03e183d344d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4aaeb87b0114f52a4a0b03e183d344d, State: Initialized, Role: FOLLOWER
I20260619 16:42:07.125348 10661 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4aaeb87b0114f52a4a0b03e183d344d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4aaeb87b0114f52a4a0b03e183d344d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34357 } }
I20260619 16:42:07.125492 10661 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4aaeb87b0114f52a4a0b03e183d344d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:42:07.125536 10661 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4aaeb87b0114f52a4a0b03e183d344d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:42:07.125602 10661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4aaeb87b0114f52a4a0b03e183d344d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:42:07.126650 10661 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4aaeb87b0114f52a4a0b03e183d344d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4aaeb87b0114f52a4a0b03e183d344d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34357 } }
I20260619 16:42:07.127115 10661 leader_election.cc:302] T 00000000000000000000000000000000 P a4aaeb87b0114f52a4a0b03e183d344d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4aaeb87b0114f52a4a0b03e183d344d; no voters:
I20260619 16:42:07.127544 10661 leader_election.cc:288] T 00000000000000000000000000000000 P a4aaeb87b0114f52a4a0b03e183d344d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:42:07.127772 10666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4aaeb87b0114f52a4a0b03e183d344d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:42:07.128206 10666 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4aaeb87b0114f52a4a0b03e183d344d [term 1 LEADER]: Becoming Leader. State: Replica: a4aaeb87b0114f52a4a0b03e183d344d, State: Running, Role: LEADER
I20260619 16:42:07.128770 10666 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4aaeb87b0114f52a4a0b03e183d344d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4aaeb87b0114f52a4a0b03e183d344d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34357 } }
I20260619 16:42:07.128990 10661 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4aaeb87b0114f52a4a0b03e183d344d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:42:07.131273 10668 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4aaeb87b0114f52a4a0b03e183d344d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4aaeb87b0114f52a4a0b03e183d344d. Latest consensus state: current_term: 1 leader_uuid: "a4aaeb87b0114f52a4a0b03e183d344d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4aaeb87b0114f52a4a0b03e183d344d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34357 } } }
I20260619 16:42:07.131285 10667 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4aaeb87b0114f52a4a0b03e183d344d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4aaeb87b0114f52a4a0b03e183d344d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4aaeb87b0114f52a4a0b03e183d344d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34357 } } }
I20260619 16:42:07.132158 10667 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4aaeb87b0114f52a4a0b03e183d344d [sys.catalog]: This master's current role is: LEADER
I20260619 16:42:07.132158 10668 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4aaeb87b0114f52a4a0b03e183d344d [sys.catalog]: This master's current role is: LEADER
W20260619 16:42:07.134460 10680 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a4aaeb87b0114f52a4a0b03e183d344d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:42:07.134580 10680 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:42:07.134706 10681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:42:07.135639 10681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:42:07.143230 10681 catalog_manager.cc:1347] Generated new cluster ID: 77ff3003773847aeab9af69aa2576365
I20260619 16:42:07.143468 10681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:42:07.158775 10681 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:42:07.160077 10681 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:42:07.169903 10681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4aaeb87b0114f52a4a0b03e183d344d: Generated new TSK 0
I20260619 16:42:07.170791 10681 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:42:07.208355 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34357
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:42:07.339987 10687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:42:07.340302 10687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:42:07.340349 10687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:42:07.344875 10687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:42:07.344967 10687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:42:07.345033 10687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:42:07.350693 10687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:42:07.352149 10687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:42:07.353345 10687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:42:07.365185 10695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:42:07.365267 10687 server_base.cc:1034] running on GCE node
W20260619 16:42:07.365185 10693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:42:07.365193 10692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:42:07.366077 10687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:42:07.367026 10687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:42:07.368258 10687 hybrid_clock.cc:648] HybridClock initialized: now 1781887327368225 us; error 49 us; skew 500 ppm
Jun 19 16:42:07 dist-test-slave-03cd krb5kdc[10580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:42:07.372330 10687 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:42:07.373790 10687 webserver.cc:460] Webserver started at http://127.0.98.1:45451/ using document root <none> and password file <none>
I20260619 16:42:07.374549 10687 fs_manager.cc:359] Metadata directory not provided
I20260619 16:42:07.374657 10687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:42:07.374949 10687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:42:07.377838 10687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-0/data/instance:
uuid: "ab0dffeedd064655af86873e342eb585"
format_stamp: "Formatted at 2026-06-19 16:42:07 on dist-test-slave-03cd"
server_key: "225fa8b3711173ee5aedfd0625b762bf"
server_key_iv: "e8f5bbd95179e233c96b968d52d1109c"
server_key_version: "encryptionkey@0"
I20260619 16:42:07.378602 10687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ab0dffeedd064655af86873e342eb585"
format_stamp: "Formatted at 2026-06-19 16:42:07 on dist-test-slave-03cd"
server_key: "225fa8b3711173ee5aedfd0625b762bf"
server_key_iv: "e8f5bbd95179e233c96b968d52d1109c"
server_key_version: "encryptionkey@0"
I20260619 16:42:07.383936 10687 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 16:42:07.387434 10702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:42:07.389321 10687 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:42:07.389536 10687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "ab0dffeedd064655af86873e342eb585"
format_stamp: "Formatted at 2026-06-19 16:42:07 on dist-test-slave-03cd"
server_key: "225fa8b3711173ee5aedfd0625b762bf"
server_key_iv: "e8f5bbd95179e233c96b968d52d1109c"
server_key_version: "encryptionkey@0"
I20260619 16:42:07.389703 10687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:42:07.403561 10687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:42:07.407980 10687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:42:07.408288 10687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:42:07.409068 10687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:42:07.410132 10687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:42:07.410207 10687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:42:07.410282 10687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:42:07.410324 10687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:42:07.422557 10687 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33297
I20260619 16:42:07.422622 10815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33297 every 8 connection(s)
I20260619 16:42:07.423805 10687 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:42:07.428158 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 10687
I20260619 16:42:07.428298 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:42:07.428743 392 external_mini_cluster.cc:1428] Setting key 087582995b3b59c470c7d72c0f9d4895
Jun 19 16:42:07 dist-test-slave-03cd krb5kdc[10580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:42:07.444310 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:07.426269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46637 (local address 127.0.98.62:34357)
0619 16:42:07.426466 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:07.426472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:07.427984 (+ 1512us) server_negotiation.cc:407] Connection header received
0619 16:42:07.429358 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:07.429363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:07.429485 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:07.429577 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:42:07.431365 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:07.432299 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:07.434491 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:07.434797 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:07.434906 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:07.438609 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:42:07.438613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:42:07.438615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:42:07.438645 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:42:07.440932 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:42:07.441859 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:42:07.441863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:42:07.441865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:42:07.442010 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:42:07.442557 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:42:07.442566 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:42:07.442570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:42:07.442957 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:42:07.443131 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 16:42:07.443895 (+ 764us) server_negotiation.cc:299] Negotiation successful
0619 16:42:07.444080 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 16:42:07.454633 10818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:07.426823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34357 (local address 127.0.98.1:46637)
0619 16:42:07.427782 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0619 16:42:07.427846 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 16:42:07.429074 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:42:07.429869 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:42:07.429878 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:42:07.430429 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:42:07.431178 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:07.431195 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:07.432495 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:07.432501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:07.434303 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:07.434319 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:07.435014 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:07.435021 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:07.435289 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:07.436188 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:42:07.436217 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:42:07.438374 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:42:07.441126 (+ 2752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:42:07.441135 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:42:07.441158 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:42:07.441717 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:42:07.442125 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:42:07.442128 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:42:07.442130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:42:07.442343 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:42:07.443162 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:42:07.443172 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:42:07.443541 (+ 369us) client_negotiation.cc:769] Sending connection context
0619 16:42:07.443803 (+ 262us) client_negotiation.cc:240] Negotiation successful
0619 16:42:07.444285 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":208,"threads_started":1}
I20260619 16:42:07.457278 10816 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34357
I20260619 16:42:07.457667 10816 heartbeater.cc:464] Registering TS with master...
I20260619 16:42:07.458312 10816 heartbeater.cc:511] Master 127.0.98.62:34357 requested a full tablet report, sending...
I20260619 16:42:07.460829 10624 ts_manager.cc:194] Registered new tserver with Master: ab0dffeedd064655af86873e342eb585 (127.0.98.1:33297)
I20260619 16:42:07.462836 10624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46637
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:42:07.472421 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34357
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:42:07.616238 10821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:42:07.616750 10821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:42:07.616842 10821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:42:07.621800 10821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:42:07.621944 10821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:42:07.622066 10821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:42:07.628794 10821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:42:07.630554 10821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:42:07.632370 10821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:42:07.645135 10821 server_base.cc:1034] running on GCE node
W20260619 16:42:07.645071 10826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:42:07.645041 10827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:42:07.645030 10829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:42:07.646039 10821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:42:07.646790 10821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:42:07.648046 10821 hybrid_clock.cc:648] HybridClock initialized: now 1781887327647967 us; error 104 us; skew 500 ppm
Jun 19 16:42:07 dist-test-slave-03cd krb5kdc[10580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:42:07.651507 10821 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:42:07.652715 10821 webserver.cc:460] Webserver started at http://127.0.98.2:42877/ using document root <none> and password file <none>
I20260619 16:42:07.653358 10821 fs_manager.cc:359] Metadata directory not provided
I20260619 16:42:07.653425 10821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:42:07.653693 10821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:42:07.656272 10821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-1/data/instance:
uuid: "77d1710531a54a96a83c3760a5fc8069"
format_stamp: "Formatted at 2026-06-19 16:42:07 on dist-test-slave-03cd"
server_key: "e23ca0711d1142903fd35e840d781d33"
server_key_iv: "e4c31449b35678327bed882702306a04"
server_key_version: "encryptionkey@0"
I20260619 16:42:07.656939 10821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-1/wal/instance:
uuid: "77d1710531a54a96a83c3760a5fc8069"
format_stamp: "Formatted at 2026-06-19 16:42:07 on dist-test-slave-03cd"
server_key: "e23ca0711d1142903fd35e840d781d33"
server_key_iv: "e4c31449b35678327bed882702306a04"
server_key_version: "encryptionkey@0"
I20260619 16:42:07.662393 10821 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:42:07.665881 10836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:42:07.667907 10821 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:42:07.668099 10821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "77d1710531a54a96a83c3760a5fc8069"
format_stamp: "Formatted at 2026-06-19 16:42:07 on dist-test-slave-03cd"
server_key: "e23ca0711d1142903fd35e840d781d33"
server_key_iv: "e4c31449b35678327bed882702306a04"
server_key_version: "encryptionkey@0"
I20260619 16:42:07.668249 10821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:42:07.682570 10821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:42:07.686476 10821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:42:07.687114 10821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:42:07.688246 10821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:42:07.689397 10821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:42:07.689487 10821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:42:07.689566 10821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:42:07.689601 10821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:42:07.700769 10821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34951
I20260619 16:42:07.700793 10949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34951 every 8 connection(s)
I20260619 16:42:07.701894 10821 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:42:07.712246 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 10821
I20260619 16:42:07.712443 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:42:07.712809 392 external_mini_cluster.cc:1428] Setting key c8168a5b373b68ba15f974ae27523719
Jun 19 16:42:07 dist-test-slave-03cd krb5kdc[10580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:42:07.722366 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:07.704684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35209 (local address 127.0.98.62:34357)
0619 16:42:07.704915 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:07.704920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:07.705923 (+ 1003us) server_negotiation.cc:407] Connection header received
0619 16:42:07.707570 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:07.707574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:07.707650 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:07.707792 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:42:07.709822 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:07.710449 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:07.712050 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:07.712309 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:07.712461 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:07.716850 (+ 4389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:42:07.716858 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:42:07.716865 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:42:07.716910 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:42:07.718839 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:42:07.719808 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:42:07.719811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:42:07.719813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:42:07.719919 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:42:07.720456 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:42:07.720460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:42:07.720461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:42:07.720756 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:42:07.720905 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 16:42:07.721848 (+ 943us) server_negotiation.cc:299] Negotiation successful
0619 16:42:07.722059 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260619 16:42:07.723400 10952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:07.705054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34357 (local address 127.0.98.2:35209)
0619 16:42:07.705666 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0619 16:42:07.705730 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 16:42:07.707274 (+ 1544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:42:07.708032 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:42:07.708041 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:42:07.708828 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:42:07.709566 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:07.709585 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:07.710632 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:07.710636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:07.711866 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:07.711881 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:07.712648 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:07.712658 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:07.712990 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:07.713826 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:42:07.713850 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:42:07.716561 (+ 2711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:42:07.718998 (+ 2437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:42:07.719010 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:42:07.719039 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:42:07.719568 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:42:07.720051 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:42:07.720057 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:42:07.720062 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:42:07.720295 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:42:07.720920 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:42:07.720932 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:42:07.721451 (+ 519us) client_negotiation.cc:769] Sending connection context
0619 16:42:07.721743 (+ 292us) client_negotiation.cc:240] Negotiation successful
0619 16:42:07.722141 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":138,"threads_started":1}
I20260619 16:42:07.725208 10950 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34357
I20260619 16:42:07.725982 10950 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:42:07.727512 10950 heartbeater.cc:511] Master 127.0.98.62:34357 requested a full tablet report, sending...
I20260619 16:42:07.729645 10624 ts_manager.cc:194] Registered new tserver with Master: 77d1710531a54a96a83c3760a5fc8069 (127.0.98.2:34951)
I20260619 16:42:07.730561 10624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:42:07.752593 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34357
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:42:07.877935 10955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:42:07.878276 10955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:42:07.878357 10955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:42:07.882630 10955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:42:07.882747 10955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:42:07.882848 10955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:42:07.888784 10955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:42:07.890038 10955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:42:07.891146 10955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:42:07.902807 10961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:42:07.902820 10963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:42:07.902807 10960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:42:07.903470 10955 server_base.cc:1034] running on GCE node
I20260619 16:42:07.904083 10955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:42:07.904847 10955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:42:07.906067 10955 hybrid_clock.cc:648] HybridClock initialized: now 1781887327906024 us; error 69 us; skew 500 ppm
Jun 19 16:42:07 dist-test-slave-03cd krb5kdc[10580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:42:07.909518 10955 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:42:07.910634 10955 webserver.cc:460] Webserver started at http://127.0.98.3:38353/ using document root <none> and password file <none>
I20260619 16:42:07.911270 10955 fs_manager.cc:359] Metadata directory not provided
I20260619 16:42:07.911362 10955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:42:07.911545 10955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:42:07.913779 10955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-2/data/instance:
uuid: "0c3253406ccd40fbb783226f04042849"
format_stamp: "Formatted at 2026-06-19 16:42:07 on dist-test-slave-03cd"
server_key: "479e25b714d6943aa248996750468407"
server_key_iv: "86274780094ce566cc16143c57a42294"
server_key_version: "encryptionkey@0"
I20260619 16:42:07.914342 10955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0c3253406ccd40fbb783226f04042849"
format_stamp: "Formatted at 2026-06-19 16:42:07 on dist-test-slave-03cd"
server_key: "479e25b714d6943aa248996750468407"
server_key_iv: "86274780094ce566cc16143c57a42294"
server_key_version: "encryptionkey@0"
I20260619 16:42:07.918793 10955 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:42:07.921869 10970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:42:07.923588 10955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:42:07.923842 10955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "0c3253406ccd40fbb783226f04042849"
format_stamp: "Formatted at 2026-06-19 16:42:07 on dist-test-slave-03cd"
server_key: "479e25b714d6943aa248996750468407"
server_key_iv: "86274780094ce566cc16143c57a42294"
server_key_version: "encryptionkey@0"
I20260619 16:42:07.923983 10955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:42:07.942683 10955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:42:07.946451 10955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:42:07.946666 10955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:42:07.947474 10955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:42:07.948820 10955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:42:07.948949 10955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 16:42:07.949046 10955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:42:07.949103 10955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:42:07.960407 10955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44679
I20260619 16:42:07.960484 11083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44679 every 8 connection(s)
I20260619 16:42:07.961462 10955 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:42:07.971776 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 10955
I20260619 16:42:07.971918 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:42:07.972369 392 external_mini_cluster.cc:1428] Setting key 6db40f9d3efcbe108862b34d7a6cae2d
Jun 19 16:42:07 dist-test-slave-03cd krb5kdc[10580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:42:07.979059 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:07.963991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53833 (local address 127.0.98.62:34357)
0619 16:42:07.964179 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:07.964184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:07.965278 (+ 1094us) server_negotiation.cc:407] Connection header received
0619 16:42:07.966925 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:07.966929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:07.966997 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:07.967115 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:42:07.968751 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:07.969342 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:07.970654 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:07.970903 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:07.971021 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:07.974976 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:42:07.974980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:42:07.974984 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:42:07.975035 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:42:07.976609 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:42:07.977290 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:42:07.977292 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:42:07.977294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:42:07.977378 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:42:07.977827 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:42:07.977829 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:42:07.977830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:42:07.978042 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:42:07.978123 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 16:42:07.978811 (+ 688us) server_negotiation.cc:299] Negotiation successful
0619 16:42:07.978932 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:42:07.980278 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:07.964391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34357 (local address 127.0.98.3:53833)
0619 16:42:07.965046 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0619 16:42:07.965116 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 16:42:07.966647 (+ 1531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:42:07.967342 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:42:07.967351 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:42:07.967966 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:42:07.968590 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:07.968606 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:07.969494 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:07.969497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:07.970517 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:07.970526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:07.971076 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:07.971079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:07.971284 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:07.972270 (+ 986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:42:07.972295 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:42:07.974749 (+ 2454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:42:07.976768 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:42:07.976774 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:42:07.976793 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:42:07.977155 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:42:07.977507 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:42:07.977510 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:42:07.977511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:42:07.977701 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:42:07.978162 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:42:07.978168 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:42:07.978547 (+ 379us) client_negotiation.cc:769] Sending connection context
0619 16:42:07.978745 (+ 198us) client_negotiation.cc:240] Negotiation successful
0619 16:42:07.979096 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":182,"threads_started":1}
I20260619 16:42:07.981990 11084 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34357
I20260619 16:42:07.982350 11084 heartbeater.cc:464] Registering TS with master...
I20260619 16:42:07.983011 11084 heartbeater.cc:511] Master 127.0.98.62:34357 requested a full tablet report, sending...
I20260619 16:42:07.984637 10624 ts_manager.cc:194] Registered new tserver with Master: 0c3253406ccd40fbb783226f04042849 (127.0.98.3:44679)
I20260619 16:42:07.985497 10624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53833
I20260619 16:42:07.987285 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:42:08.000085 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:07.989905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36066 (local address 127.0.98.62:34357)
0619 16:42:07.990056 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:07.990061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:07.990413 (+ 352us) server_negotiation.cc:407] Connection header received
0619 16:42:07.990591 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:07.990595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:07.990652 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:07.990743 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:42:07.991542 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:07.992201 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:07.993107 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:07.993363 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:07.993482 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:07.995610 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:42:07.995614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:42:07.995616 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:42:07.995716 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:42:07.997648 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:42:07.998309 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:42:07.998314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:42:07.998316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:42:07.998438 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:42:07.998870 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:42:07.998872 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:42:07.998874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:42:07.999175 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:42:07.999303 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 16:42:07.999614 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 16:42:07.999857 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:42:08.005251 10624 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:42:08.009925 10624 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:42:08.030681 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:08.022608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33297 (local address 127.0.0.1:49326)
0619 16:42:08.023329 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0619 16:42:08.023393 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 16:42:08.023530 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:42:08.024909 (+ 1379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:42:08.024913 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:42:08.024932 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:42:08.025026 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:08.025032 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:08.026504 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:08.026508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:08.028326 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:08.028345 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:08.029878 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:08.029891 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:08.030061 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:08.030120 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 16:42:08.030250 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 16:42:08.030469 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":54,"thread_start_us":153,"threads_started":1}
I20260619 16:42:08.030855 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:08.024226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45048 (local address 127.0.98.2:34951)
0619 16:42:08.024828 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:08.024838 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:08.024886 (+ 48us) server_negotiation.cc:407] Connection header received
0619 16:42:08.024963 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:08.024968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:08.025175 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:08.025325 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:42:08.025524 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:08.026464 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:08.029084 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:08.029932 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:08.030040 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:08.030209 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 16:42:08.030569 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 16:42:08.030664 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":128,"threads_started":1}
I20260619 16:42:08.030915 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:08.023648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34951 (local address 127.0.0.1:45048)
0619 16:42:08.024100 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 16:42:08.024114 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:42:08.024204 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:42:08.025311 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:42:08.025314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:42:08.025331 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:42:08.025420 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:08.025426 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:08.026619 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:08.026622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:08.028917 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:08.028930 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:08.030282 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:08.030287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:08.030390 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:08.030421 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 16:42:08.030532 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 16:42:08.030641 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":318,"thread_start_us":126,"threads_started":1}
I20260619 16:42:08.031441 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:08.022936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49326 (local address 127.0.98.1:33297)
0619 16:42:08.024135 (+ 1199us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:08.024143 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:08.024174 (+ 31us) server_negotiation.cc:407] Connection header received
0619 16:42:08.024288 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:08.024294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:08.024696 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:08.024940 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:42:08.025133 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:08.026343 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:08.028571 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:08.029616 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:08.030444 (+ 828us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:08.030688 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 16:42:08.030756 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 16:42:08.030892 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1077,"thread_start_us":137,"threads_started":1}
I20260619 16:42:08.031927 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:08.022608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44679 (local address 127.0.0.1:48306)
0619 16:42:08.023693 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0619 16:42:08.023711 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:42:08.023849 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:42:08.024452 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:42:08.024458 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:42:08.024480 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:42:08.024649 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:08.024657 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:08.026504 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:08.026508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:08.028432 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:08.028490 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:08.030109 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:08.030112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:08.030247 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:08.030269 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 16:42:08.030328 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 16:42:08.031792 (+ 1464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":973,"mutex_wait_us":672,"thread_start_us":112,"threads_started":1}
I20260619 16:42:08.032742 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:08.022749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48306 (local address 127.0.98.3:44679)
0619 16:42:08.023868 (+ 1119us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:08.023883 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:08.023911 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:42:08.023997 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:08.024004 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:08.024277 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:08.024462 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:42:08.024864 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:08.026268 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:08.028633 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:08.029659 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:08.029776 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:08.032345 (+ 2569us) server_negotiation.cc:1035] Waiting for connection context
0619 16:42:08.032417 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 16:42:08.032555 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1010,"thread_start_us":208,"threads_started":1}
I20260619 16:42:08.034881 10884 tablet_service.cc:1467] Processing CreateTablet for tablet ba0cc054151a42e5a3e5a89e4632c782 (DEFAULT_TABLE table=test-table [id=8331fe22cafa4fdbb53e783d0422e968]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:42:08.034881 10750 tablet_service.cc:1467] Processing CreateTablet for tablet ba0cc054151a42e5a3e5a89e4632c782 (DEFAULT_TABLE table=test-table [id=8331fe22cafa4fdbb53e783d0422e968]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:42:08.035818 11018 tablet_service.cc:1467] Processing CreateTablet for tablet ba0cc054151a42e5a3e5a89e4632c782 (DEFAULT_TABLE table=test-table [id=8331fe22cafa4fdbb53e783d0422e968]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:42:08.036442 10884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba0cc054151a42e5a3e5a89e4632c782. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:42:08.036542 10750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba0cc054151a42e5a3e5a89e4632c782. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:42:08.037428 11018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba0cc054151a42e5a3e5a89e4632c782. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:42:08.044926 11101 tablet_bootstrap.cc:492] T ba0cc054151a42e5a3e5a89e4632c782 P ab0dffeedd064655af86873e342eb585: Bootstrap starting.
I20260619 16:42:08.045491 11102 tablet_bootstrap.cc:492] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069: Bootstrap starting.
I20260619 16:42:08.047246 11103 tablet_bootstrap.cc:492] T ba0cc054151a42e5a3e5a89e4632c782 P 0c3253406ccd40fbb783226f04042849: Bootstrap starting.
I20260619 16:42:08.048154 11102 tablet_bootstrap.cc:654] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069: Neither blocks nor log segments found. Creating new log.
I20260619 16:42:08.048852 11101 tablet_bootstrap.cc:654] T ba0cc054151a42e5a3e5a89e4632c782 P ab0dffeedd064655af86873e342eb585: Neither blocks nor log segments found. Creating new log.
I20260619 16:42:08.049616 11102 log.cc:824] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069: Log is configured to *not* fsync() on all Append() calls
I20260619 16:42:08.049823 11101 log.cc:824] T ba0cc054151a42e5a3e5a89e4632c782 P ab0dffeedd064655af86873e342eb585: Log is configured to *not* fsync() on all Append() calls
I20260619 16:42:08.050794 11103 tablet_bootstrap.cc:654] T ba0cc054151a42e5a3e5a89e4632c782 P 0c3253406ccd40fbb783226f04042849: Neither blocks nor log segments found. Creating new log.
I20260619 16:42:08.051549 11103 log.cc:824] T ba0cc054151a42e5a3e5a89e4632c782 P 0c3253406ccd40fbb783226f04042849: Log is configured to *not* fsync() on all Append() calls
I20260619 16:42:08.052373 11102 tablet_bootstrap.cc:492] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069: No bootstrap required, opened a new log
I20260619 16:42:08.052371 11101 tablet_bootstrap.cc:492] T ba0cc054151a42e5a3e5a89e4632c782 P ab0dffeedd064655af86873e342eb585: No bootstrap required, opened a new log
I20260619 16:42:08.052614 11102 ts_tablet_manager.cc:1397] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 16:42:08.052614 11101 ts_tablet_manager.cc:1397] T ba0cc054151a42e5a3e5a89e4632c782 P ab0dffeedd064655af86873e342eb585: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 16:42:08.053714 11103 tablet_bootstrap.cc:492] T ba0cc054151a42e5a3e5a89e4632c782 P 0c3253406ccd40fbb783226f04042849: No bootstrap required, opened a new log
I20260619 16:42:08.053946 11103 ts_tablet_manager.cc:1397] T ba0cc054151a42e5a3e5a89e4632c782 P 0c3253406ccd40fbb783226f04042849: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 16:42:08.057518 11103 raft_consensus.cc:348] T ba0cc054151a42e5a3e5a89e4632c782 P 0c3253406ccd40fbb783226f04042849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c3253406ccd40fbb783226f04042849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44679 } } peers { permanent_uuid: "ab0dffeedd064655af86873e342eb585" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33297 } } peers { permanent_uuid: "77d1710531a54a96a83c3760a5fc8069" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34951 } }
I20260619 16:42:08.057464 11102 raft_consensus.cc:348] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c3253406ccd40fbb783226f04042849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44679 } } peers { permanent_uuid: "ab0dffeedd064655af86873e342eb585" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33297 } } peers { permanent_uuid: "77d1710531a54a96a83c3760a5fc8069" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34951 } }
I20260619 16:42:08.057827 11103 raft_consensus.cc:374] T ba0cc054151a42e5a3e5a89e4632c782 P 0c3253406ccd40fbb783226f04042849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:42:08.057880 11103 raft_consensus.cc:729] T ba0cc054151a42e5a3e5a89e4632c782 P 0c3253406ccd40fbb783226f04042849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c3253406ccd40fbb783226f04042849, State: Initialized, Role: FOLLOWER
I20260619 16:42:08.057886 11102 raft_consensus.cc:374] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:42:08.057727 11101 raft_consensus.cc:348] T ba0cc054151a42e5a3e5a89e4632c782 P ab0dffeedd064655af86873e342eb585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c3253406ccd40fbb783226f04042849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44679 } } peers { permanent_uuid: "ab0dffeedd064655af86873e342eb585" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33297 } } peers { permanent_uuid: "77d1710531a54a96a83c3760a5fc8069" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34951 } }
I20260619 16:42:08.057961 11101 raft_consensus.cc:374] T ba0cc054151a42e5a3e5a89e4632c782 P ab0dffeedd064655af86873e342eb585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:42:08.057953 11102 raft_consensus.cc:729] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77d1710531a54a96a83c3760a5fc8069, State: Initialized, Role: FOLLOWER
I20260619 16:42:08.058015 11101 raft_consensus.cc:729] T ba0cc054151a42e5a3e5a89e4632c782 P ab0dffeedd064655af86873e342eb585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab0dffeedd064655af86873e342eb585, State: Initialized, Role: FOLLOWER
I20260619 16:42:08.058410 11103 consensus_queue.cc:260] T ba0cc054151a42e5a3e5a89e4632c782 P 0c3253406ccd40fbb783226f04042849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c3253406ccd40fbb783226f04042849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44679 } } peers { permanent_uuid: "ab0dffeedd064655af86873e342eb585" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33297 } } peers { permanent_uuid: "77d1710531a54a96a83c3760a5fc8069" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34951 } }
I20260619 16:42:08.058518 11102 consensus_queue.cc:260] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c3253406ccd40fbb783226f04042849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44679 } } peers { permanent_uuid: "ab0dffeedd064655af86873e342eb585" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33297 } } peers { permanent_uuid: "77d1710531a54a96a83c3760a5fc8069" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34951 } }
I20260619 16:42:08.058518 11101 consensus_queue.cc:260] T ba0cc054151a42e5a3e5a89e4632c782 P ab0dffeedd064655af86873e342eb585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c3253406ccd40fbb783226f04042849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44679 } } peers { permanent_uuid: "ab0dffeedd064655af86873e342eb585" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33297 } } peers { permanent_uuid: "77d1710531a54a96a83c3760a5fc8069" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34951 } }
I20260619 16:42:08.059137 11084 heartbeater.cc:503] Master 127.0.98.62:34357 was elected leader, sending a full tablet report...
I20260619 16:42:08.059542 11103 ts_tablet_manager.cc:1428] T ba0cc054151a42e5a3e5a89e4632c782 P 0c3253406ccd40fbb783226f04042849: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 16:42:08.059741 11102 ts_tablet_manager.cc:1428] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260619 16:42:08.059741 11101 ts_tablet_manager.cc:1428] T ba0cc054151a42e5a3e5a89e4632c782 P ab0dffeedd064655af86873e342eb585: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260619 16:42:08.060173 10816 heartbeater.cc:503] Master 127.0.98.62:34357 was elected leader, sending a full tablet report...
I20260619 16:42:08.060578 10950 heartbeater.cc:503] Master 127.0.98.62:34357 was elected leader, sending a full tablet report...
I20260619 16:42:08.139250 11109 raft_consensus.cc:482] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:42:08.139587 11109 raft_consensus.cc:504] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c3253406ccd40fbb783226f04042849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44679 } } peers { permanent_uuid: "ab0dffeedd064655af86873e342eb585" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33297 } } peers { permanent_uuid: "77d1710531a54a96a83c3760a5fc8069" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34951 } }
I20260619 16:42:08.141201 11109 leader_election.cc:288] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c3253406ccd40fbb783226f04042849 (127.0.98.3:44679), ab0dffeedd064655af86873e342eb585 (127.0.98.1:33297)
I20260619 16:42:08.146739 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:08.141808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59987 (local address 127.0.98.1:33297)
0619 16:42:08.141964 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:08.141971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:08.142309 (+ 338us) server_negotiation.cc:407] Connection header received
0619 16:42:08.142386 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:08.142388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:08.142434 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:08.142504 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:42:08.142794 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:08.143510 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:08.145261 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:08.145960 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:08.146091 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:08.146173 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 16:42:08.146491 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 16:42:08.146576 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:42:08.146739 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:08.141819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53613 (local address 127.0.98.3:44679)
0619 16:42:08.141974 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:08.141979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:08.141999 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:42:08.142070 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:08.142073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:08.142124 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:08.142204 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:42:08.142610 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:08.143407 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:08.145261 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:08.145960 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:08.146091 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:08.146168 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 16:42:08.146491 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 16:42:08.146573 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:42:08.146780 10952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:08.141509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44679 (local address 127.0.98.2:53613)
0619 16:42:08.141731 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 16:42:08.141751 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:42:08.141873 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:42:08.142267 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:42:08.142270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:42:08.142294 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:42:08.142449 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:08.142455 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:08.143527 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:08.143530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:08.145085 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:08.145099 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:08.146126 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:08.146130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:08.146311 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:08.146357 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 16:42:08.146436 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 16:42:08.146563 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":209,"spinlock_wait_cycles":3456}
I20260619 16:42:08.146790 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:08.141663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33297 (local address 127.0.98.2:59987)
0619 16:42:08.142153 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 16:42:08.142164 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:42:08.142288 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:42:08.142560 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:42:08.142562 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:42:08.142582 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:42:08.142685 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:08.142689 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:08.143687 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:08.143690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:08.145095 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:42:08.145100 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:42:08.146126 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:42:08.146130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:42:08.146295 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:08.146354 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 16:42:08.146436 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 16:42:08.146562 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":90,"thread_start_us":148,"threads_started":1}
I20260619 16:42:08.147809 11038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba0cc054151a42e5a3e5a89e4632c782" candidate_uuid: "77d1710531a54a96a83c3760a5fc8069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c3253406ccd40fbb783226f04042849" is_pre_election: true
I20260619 16:42:08.147836 10770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba0cc054151a42e5a3e5a89e4632c782" candidate_uuid: "77d1710531a54a96a83c3760a5fc8069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab0dffeedd064655af86873e342eb585" is_pre_election: true
I20260619 16:42:08.148168 10770 raft_consensus.cc:2454] T ba0cc054151a42e5a3e5a89e4632c782 P ab0dffeedd064655af86873e342eb585 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77d1710531a54a96a83c3760a5fc8069 in term 0.
I20260619 16:42:08.148164 11038 raft_consensus.cc:2454] T ba0cc054151a42e5a3e5a89e4632c782 P 0c3253406ccd40fbb783226f04042849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77d1710531a54a96a83c3760a5fc8069 in term 0.
I20260619 16:42:08.148800 10839 leader_election.cc:302] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0c3253406ccd40fbb783226f04042849, 77d1710531a54a96a83c3760a5fc8069, ab0dffeedd064655af86873e342eb585; no voters:
I20260619 16:42:08.149152 11109 raft_consensus.cc:2781] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:42:08.149293 11109 raft_consensus.cc:482] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:42:08.149356 11109 raft_consensus.cc:3037] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:42:08.150836 11109 raft_consensus.cc:504] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c3253406ccd40fbb783226f04042849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44679 } } peers { permanent_uuid: "ab0dffeedd064655af86873e342eb585" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33297 } } peers { permanent_uuid: "77d1710531a54a96a83c3760a5fc8069" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34951 } }
I20260619 16:42:08.151338 11109 leader_election.cc:288] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069 [CANDIDATE]: Term 1 election: Requested vote from peers 0c3253406ccd40fbb783226f04042849 (127.0.98.3:44679), ab0dffeedd064655af86873e342eb585 (127.0.98.1:33297)
I20260619 16:42:08.151885 10770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba0cc054151a42e5a3e5a89e4632c782" candidate_uuid: "77d1710531a54a96a83c3760a5fc8069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab0dffeedd064655af86873e342eb585"
I20260619 16:42:08.151986 11038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba0cc054151a42e5a3e5a89e4632c782" candidate_uuid: "77d1710531a54a96a83c3760a5fc8069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c3253406ccd40fbb783226f04042849"
I20260619 16:42:08.152081 10770 raft_consensus.cc:3037] T ba0cc054151a42e5a3e5a89e4632c782 P ab0dffeedd064655af86873e342eb585 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:42:08.152206 11038 raft_consensus.cc:3037] T ba0cc054151a42e5a3e5a89e4632c782 P 0c3253406ccd40fbb783226f04042849 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:42:08.153394 10770 raft_consensus.cc:2454] T ba0cc054151a42e5a3e5a89e4632c782 P ab0dffeedd064655af86873e342eb585 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77d1710531a54a96a83c3760a5fc8069 in term 1.
I20260619 16:42:08.153548 11038 raft_consensus.cc:2454] T ba0cc054151a42e5a3e5a89e4632c782 P 0c3253406ccd40fbb783226f04042849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77d1710531a54a96a83c3760a5fc8069 in term 1.
I20260619 16:42:08.153869 10837 leader_election.cc:302] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77d1710531a54a96a83c3760a5fc8069, ab0dffeedd064655af86873e342eb585; no voters:
I20260619 16:42:08.154099 11109 raft_consensus.cc:2781] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:42:08.154389 11109 raft_consensus.cc:686] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069 [term 1 LEADER]: Becoming Leader. State: Replica: 77d1710531a54a96a83c3760a5fc8069, State: Running, Role: LEADER
I20260619 16:42:08.154822 11109 consensus_queue.cc:237] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c3253406ccd40fbb783226f04042849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44679 } } peers { permanent_uuid: "ab0dffeedd064655af86873e342eb585" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33297 } } peers { permanent_uuid: "77d1710531a54a96a83c3760a5fc8069" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34951 } }
I20260619 16:42:08.159061 10620 catalog_manager.cc:5495] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77d1710531a54a96a83c3760a5fc8069 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "77d1710531a54a96a83c3760a5fc8069" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c3253406ccd40fbb783226f04042849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab0dffeedd064655af86873e342eb585" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77d1710531a54a96a83c3760a5fc8069" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34951 } health_report { overall_health: HEALTHY } } }
W20260619 16:42:08.175572 10817 tablet.cc:2369] T ba0cc054151a42e5a3e5a89e4632c782 P ab0dffeedd064655af86873e342eb585: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:42:08.191057 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:08.186555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45058 (local address 127.0.98.2:34951)
0619 16:42:08.186745 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:08.186750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:08.186765 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:42:08.186983 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:08.186986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:08.187049 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:08.187182 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:42:08.187491 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:08.188341 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:08.189403 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:08.189636 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:08.189752 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:08.190108 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:42:08.190600 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:42:08.190734 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 16:42:08.190811 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 16:42:08.190877 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 16:42:08.198787 10770 raft_consensus.cc:1261] T ba0cc054151a42e5a3e5a89e4632c782 P ab0dffeedd064655af86873e342eb585 [term 1 FOLLOWER]: Refusing update from remote peer 77d1710531a54a96a83c3760a5fc8069: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:42:08.199033 11038 raft_consensus.cc:1261] T ba0cc054151a42e5a3e5a89e4632c782 P 0c3253406ccd40fbb783226f04042849 [term 1 FOLLOWER]: Refusing update from remote peer 77d1710531a54a96a83c3760a5fc8069: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:42:08.199875 11109 consensus_queue.cc:1035] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab0dffeedd064655af86873e342eb585" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:42:08.200193 11111 consensus_queue.cc:1035] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c3253406ccd40fbb783226f04042849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:42:08.203292 10951 tablet.cc:2369] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:42:08.212596 11085 tablet.cc:2369] T ba0cc054151a42e5a3e5a89e4632c782 P 0c3253406ccd40fbb783226f04042849: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:42:08.221491 10620 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:42:08.221724 10620 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:42:08.224812 10620 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4aaeb87b0114f52a4a0b03e183d344d: Sending DeleteTablet for 3 replicas of tablet ba0cc054151a42e5a3e5a89e4632c782
I20260619 16:42:08.226051 10750 tablet_service.cc:1514] Processing DeleteTablet for tablet ba0cc054151a42e5a3e5a89e4632c782 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:42:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49326
I20260619 16:42:08.226058 11018 tablet_service.cc:1514] Processing DeleteTablet for tablet ba0cc054151a42e5a3e5a89e4632c782 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:42:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48306
I20260619 16:42:08.226534 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 10687
I20260619 16:42:08.227842 10884 tablet_service.cc:1514] Processing DeleteTablet for tablet ba0cc054151a42e5a3e5a89e4632c782 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:42:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45048
I20260619 16:42:08.228513 11124 tablet_replica.cc:330] T ba0cc054151a42e5a3e5a89e4632c782 P 0c3253406ccd40fbb783226f04042849: stopping tablet replica
I20260619 16:42:08.228511 11126 tablet_replica.cc:330] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069: stopping tablet replica
I20260619 16:42:08.229036 11124 raft_consensus.cc:2229] T ba0cc054151a42e5a3e5a89e4632c782 P 0c3253406ccd40fbb783226f04042849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:42:08.229036 11126 raft_consensus.cc:2229] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:42:08.229548 11124 raft_consensus.cc:2258] T ba0cc054151a42e5a3e5a89e4632c782 P 0c3253406ccd40fbb783226f04042849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:42:08.229573 11126 raft_consensus.cc:2258] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:42:08.231652 11126 ts_tablet_manager.cc:1905] T ba0cc054151a42e5a3e5a89e4632c782 P 77d1710531a54a96a83c3760a5fc8069: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:42:08.235575 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 10821
W20260619 16:42:08.235567 10609 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33297 (error 108)
I20260619 16:42:08.238615 11124 ts_tablet_manager.cc:1905] T ba0cc054151a42e5a3e5a89e4632c782 P 0c3253406ccd40fbb783226f04042849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:42:08.238652 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:08.238197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33297 (local address 127.0.0.1:49332)
0619 16:42:08.238413 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 16:42:08.238520 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260619 16:42:08.238960 10609 catalog_manager.cc:4584] TS ab0dffeedd064655af86873e342eb585 (127.0.98.1:33297): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ba0cc054151a42e5a3e5a89e4632c782: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33297: connect: Connection refused (error 111)
I20260619 16:42:08.242221 11124 ts_tablet_manager.cc:1918] T ba0cc054151a42e5a3e5a89e4632c782 P 0c3253406ccd40fbb783226f04042849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:42:08.242383 11124 log.cc:1196] T ba0cc054151a42e5a3e5a89e4632c782 P 0c3253406ccd40fbb783226f04042849: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887259574980-392-0/minicluster-data/ts-2/wal/wals/ba0cc054151a42e5a3e5a89e4632c782
I20260619 16:42:08.242882 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 10955
I20260619 16:42:08.243312 11124 ts_tablet_manager.cc:1939] T ba0cc054151a42e5a3e5a89e4632c782 P 0c3253406ccd40fbb783226f04042849: Deleting consensus metadata
I20260619 16:42:08.243938 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:08.243227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34951 (local address 127.0.0.1:45064)
0619 16:42:08.243762 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 16:42:08.243829 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414}
I20260619 16:42:08.252140 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 10593
2026-06-19T16:42:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2560 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:42:08 dist-test-slave-03cd krb5kdc[11130](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:42:08 dist-test-slave-03cd krb5kdc[11130](info): set up 2 sockets
Jun 19 16:42:08 dist-test-slave-03cd krb5kdc[11130](info): commencing operation
krb5kdc: starting...
W20260619 16:42:09.357445 392 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:42:09 dist-test-slave-03cd krb5kdc[11130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:42:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:42:09Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260619 16:42:09.564109 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task8rFIPy/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/postgres -l logfile start
2026-06-19 16:42:12.748 UTC [11160] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 16:42:12.749 UTC [11160] LOG: listening on IPv4 address "127.0.98.20", port 54679
2026-06-19 16:42:12.754 UTC [11160] LOG: listening on Unix socket "/tmp/.s.PGSQL.54679"
2026-06-19 16:42:12.781 UTC [11163] LOG: database system was shut down at 2026-06-19 16:42:10 UTC
2026-06-19 16:42:12.787 UTC [11160] LOG: database system is ready to accept connections
I20260619 16:42:13.794474 392 mini_postgres.cc:96] Postgres bound to 54679
2026-06-19 16:42:13.804 UTC [11171] FATAL: database "slave" does not exist
127.0.98.20:54679 - accepting connections
I20260619 16:42:13.805212 392 mini_ranger.cc:161] Starting Ranger...
I20260619 16:42:13.825367 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 16:42:14.589100 392 mini_ranger.cc:87] Created ranger Postgres database
I20260619 16:42:14.589241 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ranger-admin
2026-06-19 16:42:14,903 [I] DB FLAVOR :POSTGRES
2026-06-19 16:42:14,904 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 16:42:14,904 [I] Checking connection..
2026-06-19 16:42:14,904 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 16:42:15,217 [I] Checking connection passed.
2026-06-19 16:42:15,218 [I] --------- Verifying version history table ---------
2026-06-19 16:42:15,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:42:15,504 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 16:42:15,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:42:15,783 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 16:42:15,783 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 16:42:15,783 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54679/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 16:42:16.062 UTC [11235] WARNING: there is no transaction in progress
2026-06-19 16:42:16,070 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 16:42:16,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:42:16,334 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 16:42:16,334 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 16:42:16,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 16:42:16,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 16:42:16,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-03cd.c.gcp-upstream.internal','N') ;"
2026-06-19 16:42:17,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 16:42:17,406 [I] Table x_portal_user does not exist in database ranger
2026-06-19 16:42:17,406 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 16:42:17,672 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 16:42:17,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 16:42:17,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 16:42:18,238 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 16:42:18,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54679/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 16:42:19.475 UTC [11343] WARNING: there is no transaction in progress
2026-06-19 16:42:19.496 UTC [11343] WARNING: there is no transaction in progress
2026-06-19 16:42:19.516 UTC [11343] WARNING: there is no transaction in progress
2026-06-19 16:42:19.535 UTC [11343] WARNING: there is no transaction in progress
2026-06-19 16:42:19.554 UTC [11343] WARNING: there is no transaction in progress
2026-06-19 16:42:19.581 UTC [11343] WARNING: there is no transaction in progress
2026-06-19 16:42:19.655 UTC [11343] WARNING: there is no transaction in progress
2026-06-19 16:42:19.665 UTC [11343] WARNING: there is no transaction in progress
2026-06-19 16:42:19.684 UTC [11343] WARNING: there is no transaction in progress
2026-06-19 16:42:19.697 UTC [11343] WARNING: there is no transaction in progress
2026-06-19 16:42:19.711 UTC [11343] WARNING: there is no transaction in progress
2026-06-19 16:42:19.730 UTC [11343] WARNING: there is no transaction in progress
2026-06-19 16:42:19.743 UTC [11343] WARNING: there is no transaction in progress
2026-06-19 16:42:19.751 UTC [11343] WARNING: there is no transaction in progress
2026-06-19 16:42:19.762 UTC [11343] WARNING: there is no transaction in progress
2026-06-19 16:42:19.773 UTC [11343] WARNING: there is no transaction in progress
2026-06-19 16:42:20,804 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 16:42:20,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 16:42:21,078 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 16:42:21,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 16:42:21,395 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 16:42:21,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 16:42:21,711 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 16:42:21,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 16:42:22,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 16:42:22,299 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-03cd.c.gcp-upstream.internal';"
2026-06-19 16:42:22,563 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 16:42:22,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 16:42:22,839 [I] DB_PATCHES have already been applied
I20260619 16:42:22.845618 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task8rFIPy/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 16:42:22.845697 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260619 16:42:22.849447 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:50311
Jun 19, 2026 4:42:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:42:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:42:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 4:42:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 4:42:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 4:42:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 4:42:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 16:42:24.076272 392 mini_ranger.cc:160] Time spent starting Ranger: real 10.271s user 0.005s sys 0.008s
Jun 19, 2026 4:42:48 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 4:42:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 19, 2026 4:42:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 4:42:48 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 4:42:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 4:42:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 4:42:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 4:42:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 4:42:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 4:42:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 4:42:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 4:42:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 4:42:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 4:42:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 4:42:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 4:42:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 4:42:48 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 4:42:49 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 16:42:49.953473 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 16:42:50 dist-test-slave-03cd krb5kdc[11130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887370, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 16:42:50.053113 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 16:42:50.066149 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 16:42:51.175467 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 16:42:51.175582 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ranger-kms
I20260619 16:42:51.175784 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:54679
2026-06-19 16:42:51,385 [I] DB FLAVOR :POSTGRES
2026-06-19 16:42:51,385 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 16:42:51,385 [I] Checking connection
2026-06-19 16:42:51,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54679/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 16:42:51,667 [I] connection success
2026-06-19 16:42:51,668 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 16:42:51,668 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 16:42:51,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54679/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 16:42:51,931 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 16:42:51,932 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 16:42:51,932 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 16:42:51,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54679/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 16:42:52,243 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 16:42:52.804409 392 mini_ranger_kms.cc:310] Created kms service
I20260619 16:42:53.014930 392 mini_ranger_kms.cc:326] Created kudu user
I20260619 16:42:53.067787 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 16:42:53 dist-test-slave-03cd krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887370, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:42:53.520567 392 mini_ranger_kms.cc:384] Added ranger policy
I20260619 16:42:53.520998 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 16:42:53.521034 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260619 16:42:53.525162 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11524
I20260619 16:42:53.525326 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:39871
Jun 19, 2026 4:42:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:42:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:42:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 4:42:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 4:42:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 4:42:55 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 16:42:55.861944 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.809s user 0.009s sys 0.179s
I20260619 16:42:55.862093 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 16:42:55.862227 392 mini_ranger_kms.cc:401] 127.0.98.20:39871/kms/v1/keys
Jun 19 16:42:56 dist-test-slave-03cd krb5kdc[11130](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781887376, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 16:42:56 dist-test-slave-03cd krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887376, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 16:42:56 dist-test-slave-03cd krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887376, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 16:42:59 dist-test-slave-03cd krb5kdc[11130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:42:59.268353 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35735
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35735
--ranger_config_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39871/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:42:59.388584 11574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:42:59.388916 11574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:42:59.388959 11574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:42:59.393306 11574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:42:59.393401 11574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:42:59.393420 11574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:42:59.393440 11574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:42:59.393455 11574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:42:59.399724 11574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:39871/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35735
--ranger_config_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:42:59.400990 11574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:42:59.402113 11574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:42:59.412585 11574 server_base.cc:1034] running on GCE node
W20260619 16:42:59.412525 11579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:42:59.412528 11580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:42:59.412525 11582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:42:59.413483 11574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:42:59.414809 11574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:42:59.418152 11574 hybrid_clock.cc:648] HybridClock initialized: now 1781887379418039 us; error 154 us; skew 500 ppm
Jun 19 16:42:59 dist-test-slave-03cd krb5kdc[11130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:42:59.425203 11574 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:42:59.426389 11574 webserver.cc:460] Webserver started at http://127.0.98.62:38865/ using document root <none> and password file <none>
I20260619 16:42:59.427032 11574 fs_manager.cc:359] Metadata directory not provided
I20260619 16:42:59.427095 11574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:42:59.427363 11574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:42:59 dist-test-slave-03cd krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:42:59.479375 11574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "19419085031646c88b90f96f8f4d5c9c"
format_stamp: "Formatted at 2026-06-19 16:42:59 on dist-test-slave-03cd"
server_key: "72d2cf2fd01308a73ef05c65b87051f6"
server_key_iv: "085a2a8898e3b8b4fb310de25b40020b"
server_key_version: "kuduclusterkey@0"
I20260619 16:42:59.480293 11574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "19419085031646c88b90f96f8f4d5c9c"
format_stamp: "Formatted at 2026-06-19 16:42:59 on dist-test-slave-03cd"
server_key: "72d2cf2fd01308a73ef05c65b87051f6"
server_key_iv: "085a2a8898e3b8b4fb310de25b40020b"
server_key_version: "kuduclusterkey@0"
I20260619 16:42:59.485739 11574 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:42:59.502295 11589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:42:59.503826 11574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:42:59.503970 11574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "19419085031646c88b90f96f8f4d5c9c"
format_stamp: "Formatted at 2026-06-19 16:42:59 on dist-test-slave-03cd"
server_key: "72d2cf2fd01308a73ef05c65b87051f6"
server_key_iv: "085a2a8898e3b8b4fb310de25b40020b"
server_key_version: "kuduclusterkey@0"
I20260619 16:42:59.504122 11574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:42:59.539923 11574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:42:59.543246 11574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:42:59.543460 11574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:42:59.552345 11574 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35735
I20260619 16:42:59.552363 11641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35735 every 8 connection(s)
I20260619 16:42:59.553375 11574 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:42:59.556363 11642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:42:59.558614 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 11574
I20260619 16:42:59.558792 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:42:59.559170 392 external_mini_cluster.cc:1428] Setting key 58f8e505fa39228d14da764f925a7bdc
I20260619 16:42:59.562935 11642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19419085031646c88b90f96f8f4d5c9c: Bootstrap starting.
I20260619 16:42:59.565811 11642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19419085031646c88b90f96f8f4d5c9c: Neither blocks nor log segments found. Creating new log.
I20260619 16:42:59.566696 11642 log.cc:824] T 00000000000000000000000000000000 P 19419085031646c88b90f96f8f4d5c9c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:42:59.569392 11642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19419085031646c88b90f96f8f4d5c9c: No bootstrap required, opened a new log
Jun 19 16:42:59 dist-test-slave-03cd krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:42:59.574622 11642 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19419085031646c88b90f96f8f4d5c9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19419085031646c88b90f96f8f4d5c9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35735 } }
I20260619 16:42:59.574961 11642 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19419085031646c88b90f96f8f4d5c9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:42:59.575021 11642 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19419085031646c88b90f96f8f4d5c9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19419085031646c88b90f96f8f4d5c9c, State: Initialized, Role: FOLLOWER
I20260619 16:42:59.575526 11642 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19419085031646c88b90f96f8f4d5c9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19419085031646c88b90f96f8f4d5c9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35735 } }
I20260619 16:42:59.575713 11642 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19419085031646c88b90f96f8f4d5c9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:42:59.575773 11642 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19419085031646c88b90f96f8f4d5c9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:42:59.575893 11642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19419085031646c88b90f96f8f4d5c9c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:42:59.576105 11645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:42:59.560722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60752 (local address 127.0.98.62:35735)
0619 16:42:59.561224 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0619 16:42:59.561241 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:42:59.561280 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:42:59.562003 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:42:59.562021 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:42:59.562383 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:42:59.562710 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:42:59.563752 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:59.565032 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:59.566273 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:42:59.566862 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:42:59.567034 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:42:59.570113 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:42:59.570121 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:42:59.570147 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:42:59.570188 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:42:59.572231 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:42:59.572817 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:42:59.572823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:42:59.572830 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:42:59.572975 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:42:59.573303 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:42:59.573309 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:42:59.573311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:42:59.573902 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:42:59.574582 (+ 680us) server_negotiation.cc:1035] Waiting for connection context
0619 16:42:59.574743 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 16:42:59.575052 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":179,"threads_started":1}
I20260619 16:42:59.577009 11642 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19419085031646c88b90f96f8f4d5c9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19419085031646c88b90f96f8f4d5c9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35735 } }
I20260619 16:42:59.577399 11642 leader_election.cc:302] T 00000000000000000000000000000000 P 19419085031646c88b90f96f8f4d5c9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19419085031646c88b90f96f8f4d5c9c; no voters:
I20260619 16:42:59.577787 11642 leader_election.cc:288] T 00000000000000000000000000000000 P 19419085031646c88b90f96f8f4d5c9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:42:59.577904 11647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19419085031646c88b90f96f8f4d5c9c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:42:59.578157 11647 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19419085031646c88b90f96f8f4d5c9c [term 1 LEADER]: Becoming Leader. State: Replica: 19419085031646c88b90f96f8f4d5c9c, State: Running, Role: LEADER
I20260619 16:42:59.579211 11647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19419085031646c88b90f96f8f4d5c9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19419085031646c88b90f96f8f4d5c9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35735 } }
I20260619 16:42:59.579221 11642 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19419085031646c88b90f96f8f4d5c9c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:42:59.580964 11648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19419085031646c88b90f96f8f4d5c9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19419085031646c88b90f96f8f4d5c9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19419085031646c88b90f96f8f4d5c9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35735 } } }
I20260619 16:42:59.581102 11648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19419085031646c88b90f96f8f4d5c9c [sys.catalog]: This master's current role is: LEADER
I20260619 16:42:59.581413 11649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19419085031646c88b90f96f8f4d5c9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19419085031646c88b90f96f8f4d5c9c. Latest consensus state: current_term: 1 leader_uuid: "19419085031646c88b90f96f8f4d5c9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19419085031646c88b90f96f8f4d5c9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35735 } } }
I20260619 16:42:59.581501 11649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19419085031646c88b90f96f8f4d5c9c [sys.catalog]: This master's current role is: LEADER
I20260619 16:42:59.581881 11642 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 16:42:59.583209 11659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:42:59.589636 11659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:42:59.598171 11659 catalog_manager.cc:1347] Generated new cluster ID: 65cfe5c1ef464b04bc82bd6ada90ca84
I20260619 16:42:59.598260 11659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:42:59.609118 11659 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:42:59.610257 11659 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:42:59.620116 11659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19419085031646c88b90f96f8f4d5c9c: Generated new TSK 0
I20260619 16:42:59.620968 11659 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:43:00.384531 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39871/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35735
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:00.544183 11687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:00.544469 11687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:00.544514 11687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:00.548875 11687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:00.548949 11687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:00.549010 11687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:43:00.557260 11687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:39871/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:43:00.558626 11687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:00.559903 11687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:00.574852 11697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:00.574885 11694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:00.575192 11687 server_base.cc:1034] running on GCE node
W20260619 16:43:00.574923 11695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:00.576037 11687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:00.576848 11687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:00.578070 11687 hybrid_clock.cc:648] HybridClock initialized: now 1781887380578036 us; error 53 us; skew 500 ppm
Jun 19 16:43:00 dist-test-slave-03cd krb5kdc[11130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:00.582322 11687 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:43:00.584008 11687 webserver.cc:460] Webserver started at http://127.0.98.1:34913/ using document root <none> and password file <none>
I20260619 16:43:00.584842 11687 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:00.584950 11687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:00.585295 11687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:43:00 dist-test-slave-03cd krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:43:00.601162 11687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-0/data/instance:
uuid: "6a513c734f9449079aae6e92747afdcd"
format_stamp: "Formatted at 2026-06-19 16:43:00 on dist-test-slave-03cd"
server_key: "92539c51c99fe2b2b4b295ea762537d8"
server_key_iv: "810a1d9646e8d9793f28326d3091e726"
server_key_version: "kuduclusterkey@0"
I20260619 16:43:00.601919 11687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6a513c734f9449079aae6e92747afdcd"
format_stamp: "Formatted at 2026-06-19 16:43:00 on dist-test-slave-03cd"
server_key: "92539c51c99fe2b2b4b295ea762537d8"
server_key_iv: "810a1d9646e8d9793f28326d3091e726"
server_key_version: "kuduclusterkey@0"
I20260619 16:43:00.613476 11687 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.004s sys 0.000s
I20260619 16:43:00.633333 11704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:00.635072 11687 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:43:00.635236 11687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "6a513c734f9449079aae6e92747afdcd"
format_stamp: "Formatted at 2026-06-19 16:43:00 on dist-test-slave-03cd"
server_key: "92539c51c99fe2b2b4b295ea762537d8"
server_key_iv: "810a1d9646e8d9793f28326d3091e726"
server_key_version: "kuduclusterkey@0"
I20260619 16:43:00.635392 11687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:00.714885 11687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:00.718871 11687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:00.719072 11687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:00.719755 11687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:00.720906 11687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:00.720970 11687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:00.721026 11687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:00.721081 11687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 16:43:00.733150 11687 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46803
I20260619 16:43:00.733361 11819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46803 every 8 connection(s)
I20260619 16:43:00.734443 11687 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:43:00.735529 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 11687
I20260619 16:43:00.735644 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:43:00.736060 392 external_mini_cluster.cc:1428] Setting key b879b67be3b5c8989e98bfc05c0f1df2
Jun 19 16:43:00 dist-test-slave-03cd krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:43:00.754118 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:00.738240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40827 (local address 127.0.98.62:35735)
0619 16:43:00.740165 (+ 1925us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:00.740170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:00.740190 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:43:00.740465 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:00.740469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:00.740524 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:00.740609 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:00.742274 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:00.743095 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:00.745220 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:00.745597 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:00.745720 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:00.749297 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:00.749302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:00.749307 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:00.749344 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:00.751375 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:00.752107 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:00.752111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:00.752113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:00.752215 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:00.752598 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:00.752601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:00.752603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:00.752895 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:00.752998 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:00.753779 (+ 781us) server_negotiation.cc:299] Negotiation successful
0619 16:43:00.753926 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1760,"thread_start_us":82,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 16:43:00.756160 11822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:00.737081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35735 (local address 127.0.98.1:40827)
0619 16:43:00.738677 (+ 1596us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:00.738741 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:00.739800 (+ 1059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:00.740925 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:00.740936 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:00.741403 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:00.742074 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:00.742094 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:00.743255 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:00.743259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:00.745068 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:00.745083 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:00.745787 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:00.745794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:00.746041 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:00.746954 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:00.746977 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:00.748991 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:00.751535 (+ 2544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:00.751545 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:00.751571 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:00.751976 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:00.752317 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:00.752322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:00.752324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:00.752489 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:00.753016 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:00.753022 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:00.753390 (+ 368us) client_negotiation.cc:769] Sending connection context
0619 16:43:00.753662 (+ 272us) client_negotiation.cc:240] Negotiation successful
0619 16:43:00.754094 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1365,"thread_start_us":158,"threads_started":1}
I20260619 16:43:00.757952 11820 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35735
I20260619 16:43:00.758224 11820 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:00.758842 11820 heartbeater.cc:511] Master 127.0.98.62:35735 requested a full tablet report, sending...
I20260619 16:43:00.760874 11606 ts_manager.cc:194] Registered new tserver with Master: 6a513c734f9449079aae6e92747afdcd (127.0.98.1:46803)
I20260619 16:43:00.762514 11606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40827
Jun 19 16:43:00 dist-test-slave-03cd krb5kdc[11130](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781887380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:43:00.781185 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39871/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35735
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:00.952981 11826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:00.953496 11826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:00.953603 11826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:00.960345 11826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:00.960440 11826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:00.960532 11826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:43:00.969195 11826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:39871/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:43:00.970729 11826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:00.972186 11826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:00.985394 11833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:00.985925 11826 server_base.cc:1034] running on GCE node
W20260619 16:43:00.985981 11832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:00.986090 11835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:00.986687 11826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:00.987496 11826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:00.988735 11826 hybrid_clock.cc:648] HybridClock initialized: now 1781887380988721 us; error 53 us; skew 500 ppm
Jun 19 16:43:00 dist-test-slave-03cd krb5kdc[11130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:00.992341 11826 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:43:00.993742 11826 webserver.cc:460] Webserver started at http://127.0.98.2:43039/ using document root <none> and password file <none>
I20260619 16:43:00.994592 11826 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:00.994714 11826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:00.995002 11826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:43:00 dist-test-slave-03cd krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:43:01.011529 11826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-1/data/instance:
uuid: "7878625f9bcf4b58bb8343e4316f3601"
format_stamp: "Formatted at 2026-06-19 16:43:01 on dist-test-slave-03cd"
server_key: "551776b3fb0b5fe9883010cb79a8cc21"
server_key_iv: "06ec2d7136901a385c6e292f0cd20a5e"
server_key_version: "kuduclusterkey@0"
I20260619 16:43:01.012341 11826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7878625f9bcf4b58bb8343e4316f3601"
format_stamp: "Formatted at 2026-06-19 16:43:01 on dist-test-slave-03cd"
server_key: "551776b3fb0b5fe9883010cb79a8cc21"
server_key_iv: "06ec2d7136901a385c6e292f0cd20a5e"
server_key_version: "kuduclusterkey@0"
I20260619 16:43:01.017653 11826 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:43:01.044626 11842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:01.046566 11826 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260619 16:43:01.046866 11826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "7878625f9bcf4b58bb8343e4316f3601"
format_stamp: "Formatted at 2026-06-19 16:43:01 on dist-test-slave-03cd"
server_key: "551776b3fb0b5fe9883010cb79a8cc21"
server_key_iv: "06ec2d7136901a385c6e292f0cd20a5e"
server_key_version: "kuduclusterkey@0"
I20260619 16:43:01.047040 11826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:01.098620 11826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:01.103229 11826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:01.103528 11826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:01.104458 11826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:01.105672 11826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:01.105798 11826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:01.105913 11826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:01.105983 11826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:01.122467 11826 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45733
I20260619 16:43:01.124765 11826 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:43:01.126309 11955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45733 every 8 connection(s)
I20260619 16:43:01.131311 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 11826
I20260619 16:43:01.131444 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:43:01.131829 392 external_mini_cluster.cc:1428] Setting key 7f3d5c99d12175c3a21a3ae15382e60b
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Jun 19 16:43:01 dist-test-slave-03cd krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 16:43:01.164824 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:01.132695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37015 (local address 127.0.98.62:35735)
0619 16:43:01.133225 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:01.133232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:01.140492 (+ 7260us) server_negotiation.cc:407] Connection header received
0619 16:43:01.141962 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:01.141967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:01.142036 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:01.142151 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:01.144238 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.145076 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.151212 (+ 6136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.151718 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.151860 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:01.159710 (+ 7850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:01.159716 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:01.159718 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:01.159771 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:01.161802 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:01.162658 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:01.162664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:01.162665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:01.162764 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:01.163264 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:01.163267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:01.163269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:01.163533 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:01.163626 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:01.164465 (+ 839us) server_negotiation.cc:299] Negotiation successful
0619 16:43:01.164623 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410}
I20260619 16:43:01.166900 11958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:01.129668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35735 (local address 127.0.98.2:37015)
0619 16:43:01.140236 (+ 10568us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:01.140342 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:01.141663 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:01.142536 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:01.142548 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:01.143085 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:01.144006 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:01.144031 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.145271 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.145275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:01.146790 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:01.146816 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.152097 (+ 5281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.152103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:01.152423 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:01.153447 (+ 1024us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:01.153474 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:01.159385 (+ 5911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:01.162021 (+ 2636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:01.162030 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:01.162053 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:01.162511 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:01.162891 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:01.162894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:01.162895 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:01.163134 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:01.163728 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:01.163736 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:01.164146 (+ 410us) client_negotiation.cc:769] Sending connection context
0619 16:43:01.165359 (+ 1213us) client_negotiation.cc:240] Negotiation successful
0619 16:43:01.165763 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10182,"thread_start_us":190,"threads_started":1}
I20260619 16:43:01.168660 11956 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35735
I20260619 16:43:01.168931 11956 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:01.169585 11956 heartbeater.cc:511] Master 127.0.98.62:35735 requested a full tablet report, sending...
I20260619 16:43:01.171092 11606 ts_manager.cc:194] Registered new tserver with Master: 7878625f9bcf4b58bb8343e4316f3601 (127.0.98.2:45733)
I20260619 16:43:01.171936 11606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:43:01.192137 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39871/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35735
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:43:01 dist-test-slave-03cd krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260619 16:43:01.414647 11961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:01.415174 11961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:01.415288 11961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:01.422622 11961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:01.422835 11961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:01.423009 11961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:43:01.432066 11961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:39871/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:43:01.433851 11961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:01.435402 11961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:01.449075 11969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:01.451036 11961 server_base.cc:1034] running on GCE node
W20260619 16:43:01.456796 11967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:01.460755 11966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:01.461480 11961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:01.462393 11961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:01.469830 11961 hybrid_clock.cc:648] HybridClock initialized: now 1781887381469775 us; error 57 us; skew 500 ppm
Jun 19 16:43:01 dist-test-slave-03cd krb5kdc[11130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:01.474318 11961 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:43:01.475771 11961 webserver.cc:460] Webserver started at http://127.0.98.3:33537/ using document root <none> and password file <none>
I20260619 16:43:01.476562 11961 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:01.476709 11961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:01.476997 11961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:43:01 dist-test-slave-03cd krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:43:01.497174 11961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-2/data/instance:
uuid: "33556bb7a26e4cad99f3656cebaca9e9"
format_stamp: "Formatted at 2026-06-19 16:43:01 on dist-test-slave-03cd"
server_key: "a050fe150a6c571e597bb7f515a2a6df"
server_key_iv: "fe5978c4e444d8c1426ed458e6b415b2"
server_key_version: "kuduclusterkey@0"
I20260619 16:43:01.499254 11961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-2/wal/instance:
uuid: "33556bb7a26e4cad99f3656cebaca9e9"
format_stamp: "Formatted at 2026-06-19 16:43:01 on dist-test-slave-03cd"
server_key: "a050fe150a6c571e597bb7f515a2a6df"
server_key_iv: "fe5978c4e444d8c1426ed458e6b415b2"
server_key_version: "kuduclusterkey@0"
I20260619 16:43:01.516052 11961 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.005s sys 0.000s
I20260619 16:43:01.552192 11976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:01.561261 11961 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.002s sys 0.000s
I20260619 16:43:01.561426 11961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "33556bb7a26e4cad99f3656cebaca9e9"
format_stamp: "Formatted at 2026-06-19 16:43:01 on dist-test-slave-03cd"
server_key: "a050fe150a6c571e597bb7f515a2a6df"
server_key_iv: "fe5978c4e444d8c1426ed458e6b415b2"
server_key_version: "kuduclusterkey@0"
I20260619 16:43:01.561570 11961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:01.613458 11961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:01.617959 11961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:01.618283 11961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:01.619103 11961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:01.620482 11961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:01.620612 11961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:01.620734 11961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:01.620815 11961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:01.645103 11961 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46465
I20260619 16:43:01.646476 11961 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:43:01.650738 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 11961
I20260619 16:43:01.650878 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:43:01.651181 392 external_mini_cluster.cc:1428] Setting key 8a7ad43f20467d3473519ddf3f888cf5
I20260619 16:43:01.657208 12089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46465 every 8 connection(s)
Jun 19 16:43:01 dist-test-slave-03cd krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:43:01.690192 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:01.659502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38127 (local address 127.0.98.62:35735)
0619 16:43:01.659639 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:01.659644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:01.662210 (+ 2566us) server_negotiation.cc:407] Connection header received
0619 16:43:01.669016 (+ 6806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:01.669022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:01.669094 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:01.669153 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:01.671069 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.671884 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.674520 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.674892 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.674980 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:01.680736 (+ 5756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:01.680741 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:01.680744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:01.680789 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:01.682892 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:01.687854 (+ 4962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:01.687860 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:01.687862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:01.688037 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:01.688632 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:01.688639 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:01.688643 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:01.688919 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:01.689024 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:01.689854 (+ 830us) server_negotiation.cc:299] Negotiation successful
0619 16:43:01.690016 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:43:01.692286 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:01.660094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35735 (local address 127.0.98.3:38127)
0619 16:43:01.662017 (+ 1923us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:01.662081 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:01.668605 (+ 6524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:01.669646 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:01.669658 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:01.670094 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:01.670863 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:01.670886 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.672060 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.672064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:01.673495 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:01.673508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.675274 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.675279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:01.675515 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:01.677779 (+ 2264us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:01.677804 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:01.679854 (+ 2050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:01.683077 (+ 3223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:01.683085 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:01.683108 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:01.683550 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:01.688211 (+ 4661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:01.688215 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:01.688217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:01.688512 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:01.689106 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:01.689115 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:01.689526 (+ 411us) client_negotiation.cc:769] Sending connection context
0619 16:43:01.690643 (+ 1117us) client_negotiation.cc:240] Negotiation successful
0619 16:43:01.691089 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1686,"spinlock_wait_cycles":46976,"thread_start_us":236,"threads_started":1}
I20260619 16:43:01.694083 12090 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35735
I20260619 16:43:01.694350 12090 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:01.694993 12090 heartbeater.cc:511] Master 127.0.98.62:35735 requested a full tablet report, sending...
I20260619 16:43:01.696601 11606 ts_manager.cc:194] Registered new tserver with Master: 33556bb7a26e4cad99f3656cebaca9e9 (127.0.98.3:46465)
I20260619 16:43:01.697362 11606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38127
I20260619 16:43:01.701357 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:43:01.738852 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:01.710525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60754 (local address 127.0.98.62:35735)
0619 16:43:01.710671 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:01.710676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:01.719373 (+ 8697us) server_negotiation.cc:407] Connection header received
0619 16:43:01.720084 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:01.720088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:01.720158 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:01.720219 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:01.724531 (+ 4312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.725363 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.729323 (+ 3960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.729673 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.729778 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:01.733683 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:01.733688 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:01.733691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:01.733732 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:01.735646 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:01.737188 (+ 1542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:01.737193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:01.737195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:01.737314 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:01.737705 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:01.737709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:01.737711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:01.737995 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:01.738124 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:01.738531 (+ 407us) server_negotiation.cc:299] Negotiation successful
0619 16:43:01.738683 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:43:01.743009 11606 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:43:01.747855 11606 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:43:01.765749 11820 heartbeater.cc:503] Master 127.0.98.62:35735 was elected leader, sending a full tablet report...
I20260619 16:43:01.775168 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:01.759958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46803 (local address 127.0.0.1:59004)
0619 16:43:01.763104 (+ 3146us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:01.764047 (+ 943us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:01.764191 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:01.765584 (+ 1393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:01.765590 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:01.765613 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:01.765769 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:01.765780 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.768980 (+ 3200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.768984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:01.771439 (+ 2455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:01.771451 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.774568 (+ 3117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.774572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:01.774715 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:01.774765 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 16:43:01.774860 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 16:43:01.775001 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2951,"thread_start_us":100,"threads_started":1}
I20260619 16:43:01.777724 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:01.760062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59004 (local address 127.0.98.1:46803)
0619 16:43:01.765039 (+ 4977us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:01.765048 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:01.765076 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:43:01.765156 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:01.765162 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:01.765407 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:01.765939 (+ 532us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:01.766094 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.768135 (+ 2041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.771603 (+ 3468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.772543 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.777165 (+ 4622us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:01.777367 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:01.777443 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 16:43:01.777556 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4859,"thread_start_us":103,"threads_started":1}
I20260619 16:43:01.778900 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:01.762485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46465 (local address 127.0.0.1:58432)
0619 16:43:01.764696 (+ 2211us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:01.764708 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:01.764773 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:01.773718 (+ 8945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:01.773721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:01.773756 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:01.773863 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:01.773869 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.775416 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.775419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:01.776897 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:01.776909 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.778164 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.778168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:01.778560 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:01.778586 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 16:43:01.778659 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 16:43:01.778737 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2160,"mutex_wait_us":246,"thread_start_us":88,"threads_started":1}
I20260619 16:43:01.779506 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:01.762634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58432 (local address 127.0.98.3:46465)
0619 16:43:01.773155 (+ 10521us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:01.773167 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:01.773198 (+ 31us) server_negotiation.cc:407] Connection header received
0619 16:43:01.773289 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:01.773296 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:01.773537 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:01.774026 (+ 489us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:01.774166 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.775276 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.777106 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.778014 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.778301 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:01.778452 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:01.779249 (+ 797us) server_negotiation.cc:299] Negotiation successful
0619 16:43:01.779365 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10383,"thread_start_us":111,"threads_started":1}
I20260619 16:43:01.782447 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:01.761774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45733 (local address 127.0.0.1:33946)
0619 16:43:01.764454 (+ 2680us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:01.764467 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:01.764538 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:01.772712 (+ 8174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:01.772715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:01.772748 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:01.772860 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:01.772866 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.778062 (+ 5196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.778066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:01.779823 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:01.779834 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.781026 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.781030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:01.781127 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:01.781152 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 16:43:01.781201 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 16:43:01.781275 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2611,"mutex_wait_us":25,"thread_start_us":113,"threads_started":1}
I20260619 16:43:01.782398 12009 tablet_service.cc:1467] Processing CreateTablet for tablet 700f8c61f3c74b908919ee4a7211da5f (DEFAULT_TABLE table=test-table [id=fc528bbb03264b2da7884b77408efc15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:01.782581 11736 tablet_service.cc:1467] Processing CreateTablet for tablet 700f8c61f3c74b908919ee4a7211da5f (DEFAULT_TABLE table=test-table [id=fc528bbb03264b2da7884b77408efc15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:01.783581 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:01.761939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33946 (local address 127.0.98.2:45733)
0619 16:43:01.768445 (+ 6506us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:01.768452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:01.768479 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:43:01.768561 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:01.768566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:01.768810 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:01.773050 (+ 4240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:01.773199 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.774388 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.779998 (+ 5610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.780875 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.783002 (+ 2127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:01.783254 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:01.783316 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 16:43:01.783438 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6380,"thread_start_us":92,"threads_started":1}
I20260619 16:43:01.783615 12009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 700f8c61f3c74b908919ee4a7211da5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:01.783630 11736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 700f8c61f3c74b908919ee4a7211da5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:01.786849 11890 tablet_service.cc:1467] Processing CreateTablet for tablet 700f8c61f3c74b908919ee4a7211da5f (DEFAULT_TABLE table=test-table [id=fc528bbb03264b2da7884b77408efc15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:01.788282 11890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 700f8c61f3c74b908919ee4a7211da5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:01.799532 12110 tablet_bootstrap.cc:492] T 700f8c61f3c74b908919ee4a7211da5f P 6a513c734f9449079aae6e92747afdcd: Bootstrap starting.
I20260619 16:43:01.802228 12110 tablet_bootstrap.cc:654] T 700f8c61f3c74b908919ee4a7211da5f P 6a513c734f9449079aae6e92747afdcd: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:01.803624 12110 log.cc:824] T 700f8c61f3c74b908919ee4a7211da5f P 6a513c734f9449079aae6e92747afdcd: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:01.804484 12111 tablet_bootstrap.cc:492] T 700f8c61f3c74b908919ee4a7211da5f P 7878625f9bcf4b58bb8343e4316f3601: Bootstrap starting.
I20260619 16:43:01.806988 12111 tablet_bootstrap.cc:654] T 700f8c61f3c74b908919ee4a7211da5f P 7878625f9bcf4b58bb8343e4316f3601: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:01.808017 12111 log.cc:824] T 700f8c61f3c74b908919ee4a7211da5f P 7878625f9bcf4b58bb8343e4316f3601: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:01.793946 12109 tablet_bootstrap.cc:492] T 700f8c61f3c74b908919ee4a7211da5f P 33556bb7a26e4cad99f3656cebaca9e9: Bootstrap starting.
I20260619 16:43:01.812333 12109 tablet_bootstrap.cc:654] T 700f8c61f3c74b908919ee4a7211da5f P 33556bb7a26e4cad99f3656cebaca9e9: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:01.812353 12110 tablet_bootstrap.cc:492] T 700f8c61f3c74b908919ee4a7211da5f P 6a513c734f9449079aae6e92747afdcd: No bootstrap required, opened a new log
I20260619 16:43:01.812588 12110 ts_tablet_manager.cc:1397] T 700f8c61f3c74b908919ee4a7211da5f P 6a513c734f9449079aae6e92747afdcd: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.002s
I20260619 16:43:01.813427 12109 log.cc:824] T 700f8c61f3c74b908919ee4a7211da5f P 33556bb7a26e4cad99f3656cebaca9e9: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:01.816020 12109 tablet_bootstrap.cc:492] T 700f8c61f3c74b908919ee4a7211da5f P 33556bb7a26e4cad99f3656cebaca9e9: No bootstrap required, opened a new log
I20260619 16:43:01.816288 12109 ts_tablet_manager.cc:1397] T 700f8c61f3c74b908919ee4a7211da5f P 33556bb7a26e4cad99f3656cebaca9e9: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.000s
I20260619 16:43:01.817389 12110 raft_consensus.cc:348] T 700f8c61f3c74b908919ee4a7211da5f P 6a513c734f9449079aae6e92747afdcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a513c734f9449079aae6e92747afdcd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46803 } } peers { permanent_uuid: "7878625f9bcf4b58bb8343e4316f3601" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45733 } } peers { permanent_uuid: "33556bb7a26e4cad99f3656cebaca9e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46465 } }
I20260619 16:43:01.817724 12110 raft_consensus.cc:374] T 700f8c61f3c74b908919ee4a7211da5f P 6a513c734f9449079aae6e92747afdcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:01.817786 12110 raft_consensus.cc:729] T 700f8c61f3c74b908919ee4a7211da5f P 6a513c734f9449079aae6e92747afdcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a513c734f9449079aae6e92747afdcd, State: Initialized, Role: FOLLOWER
I20260619 16:43:01.818327 12110 consensus_queue.cc:260] T 700f8c61f3c74b908919ee4a7211da5f P 6a513c734f9449079aae6e92747afdcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a513c734f9449079aae6e92747afdcd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46803 } } peers { permanent_uuid: "7878625f9bcf4b58bb8343e4316f3601" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45733 } } peers { permanent_uuid: "33556bb7a26e4cad99f3656cebaca9e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46465 } }
I20260619 16:43:01.819546 12110 ts_tablet_manager.cc:1428] T 700f8c61f3c74b908919ee4a7211da5f P 6a513c734f9449079aae6e92747afdcd: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260619 16:43:01.821412 12111 tablet_bootstrap.cc:492] T 700f8c61f3c74b908919ee4a7211da5f P 7878625f9bcf4b58bb8343e4316f3601: No bootstrap required, opened a new log
I20260619 16:43:01.821645 12111 ts_tablet_manager.cc:1397] T 700f8c61f3c74b908919ee4a7211da5f P 7878625f9bcf4b58bb8343e4316f3601: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20260619 16:43:01.821676 12109 raft_consensus.cc:348] T 700f8c61f3c74b908919ee4a7211da5f P 33556bb7a26e4cad99f3656cebaca9e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a513c734f9449079aae6e92747afdcd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46803 } } peers { permanent_uuid: "7878625f9bcf4b58bb8343e4316f3601" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45733 } } peers { permanent_uuid: "33556bb7a26e4cad99f3656cebaca9e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46465 } }
I20260619 16:43:01.822091 12109 raft_consensus.cc:374] T 700f8c61f3c74b908919ee4a7211da5f P 33556bb7a26e4cad99f3656cebaca9e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:01.822180 12109 raft_consensus.cc:729] T 700f8c61f3c74b908919ee4a7211da5f P 33556bb7a26e4cad99f3656cebaca9e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33556bb7a26e4cad99f3656cebaca9e9, State: Initialized, Role: FOLLOWER
I20260619 16:43:01.822789 12109 consensus_queue.cc:260] T 700f8c61f3c74b908919ee4a7211da5f P 33556bb7a26e4cad99f3656cebaca9e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a513c734f9449079aae6e92747afdcd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46803 } } peers { permanent_uuid: "7878625f9bcf4b58bb8343e4316f3601" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45733 } } peers { permanent_uuid: "33556bb7a26e4cad99f3656cebaca9e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46465 } }
I20260619 16:43:01.824442 12109 ts_tablet_manager.cc:1428] T 700f8c61f3c74b908919ee4a7211da5f P 33556bb7a26e4cad99f3656cebaca9e9: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260619 16:43:01.824859 12090 heartbeater.cc:503] Master 127.0.98.62:35735 was elected leader, sending a full tablet report...
Jun 19 16:43:01 dist-test-slave-03cd krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:43:01.830974 12111 raft_consensus.cc:348] T 700f8c61f3c74b908919ee4a7211da5f P 7878625f9bcf4b58bb8343e4316f3601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a513c734f9449079aae6e92747afdcd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46803 } } peers { permanent_uuid: "7878625f9bcf4b58bb8343e4316f3601" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45733 } } peers { permanent_uuid: "33556bb7a26e4cad99f3656cebaca9e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46465 } }
I20260619 16:43:01.837785 12111 raft_consensus.cc:374] T 700f8c61f3c74b908919ee4a7211da5f P 7878625f9bcf4b58bb8343e4316f3601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:01.837963 12111 raft_consensus.cc:729] T 700f8c61f3c74b908919ee4a7211da5f P 7878625f9bcf4b58bb8343e4316f3601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7878625f9bcf4b58bb8343e4316f3601, State: Initialized, Role: FOLLOWER
I20260619 16:43:01.841850 12111 consensus_queue.cc:260] T 700f8c61f3c74b908919ee4a7211da5f P 7878625f9bcf4b58bb8343e4316f3601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a513c734f9449079aae6e92747afdcd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46803 } } peers { permanent_uuid: "7878625f9bcf4b58bb8343e4316f3601" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45733 } } peers { permanent_uuid: "33556bb7a26e4cad99f3656cebaca9e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46465 } }
I20260619 16:43:01.843377 12111 ts_tablet_manager.cc:1428] T 700f8c61f3c74b908919ee4a7211da5f P 7878625f9bcf4b58bb8343e4316f3601: Time spent starting tablet: real 0.022s user 0.004s sys 0.005s
I20260619 16:43:01.843950 11956 heartbeater.cc:503] Master 127.0.98.62:35735 was elected leader, sending a full tablet report...
I20260619 16:43:01.877594 12117 raft_consensus.cc:482] T 700f8c61f3c74b908919ee4a7211da5f P 7878625f9bcf4b58bb8343e4316f3601 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:43:01.877861 12117 raft_consensus.cc:504] T 700f8c61f3c74b908919ee4a7211da5f P 7878625f9bcf4b58bb8343e4316f3601 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a513c734f9449079aae6e92747afdcd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46803 } } peers { permanent_uuid: "7878625f9bcf4b58bb8343e4316f3601" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45733 } } peers { permanent_uuid: "33556bb7a26e4cad99f3656cebaca9e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46465 } }
I20260619 16:43:01.879420 12117 leader_election.cc:288] T 700f8c61f3c74b908919ee4a7211da5f P 7878625f9bcf4b58bb8343e4316f3601 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a513c734f9449079aae6e92747afdcd (127.0.98.1:46803), 33556bb7a26e4cad99f3656cebaca9e9 (127.0.98.3:46465)
W20260619 16:43:01.879516 11957 tablet.cc:2369] T 700f8c61f3c74b908919ee4a7211da5f P 7878625f9bcf4b58bb8343e4316f3601: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:43:01.890924 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:01.880838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37921 (local address 127.0.98.1:46803)
0619 16:43:01.880966 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:01.880971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:01.880986 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:43:01.881051 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:01.881054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:01.881107 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:01.881184 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:01.881799 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.882834 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.888020 (+ 5186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.888739 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.888867 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:01.888949 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:01.890657 (+ 1708us) server_negotiation.cc:299] Negotiation successful
0619 16:43:01.890769 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 16:43:01.891481 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:01.879895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46803 (local address 127.0.98.2:37921)
0619 16:43:01.880375 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:01.880392 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:01.880507 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:01.881462 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:01.881465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:01.881488 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:01.881644 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:01.881653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.884304 (+ 2651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.884308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:01.887789 (+ 3481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:01.887825 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.890347 (+ 2522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.890351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:01.890477 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:01.890512 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 16:43:01.891202 (+ 690us) client_negotiation.cc:240] Negotiation successful
0619 16:43:01.891316 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":132,"threads_started":1}
I20260619 16:43:01.892303 11774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "700f8c61f3c74b908919ee4a7211da5f" candidate_uuid: "7878625f9bcf4b58bb8343e4316f3601" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a513c734f9449079aae6e92747afdcd" is_pre_election: true
I20260619 16:43:01.892711 11774 raft_consensus.cc:2454] T 700f8c61f3c74b908919ee4a7211da5f P 6a513c734f9449079aae6e92747afdcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7878625f9bcf4b58bb8343e4316f3601 in term 0.
I20260619 16:43:01.893368 11844 leader_election.cc:302] T 700f8c61f3c74b908919ee4a7211da5f P 7878625f9bcf4b58bb8343e4316f3601 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a513c734f9449079aae6e92747afdcd, 7878625f9bcf4b58bb8343e4316f3601; no voters:
I20260619 16:43:01.893712 12117 raft_consensus.cc:2781] T 700f8c61f3c74b908919ee4a7211da5f P 7878625f9bcf4b58bb8343e4316f3601 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:43:01.893792 12117 raft_consensus.cc:482] T 700f8c61f3c74b908919ee4a7211da5f P 7878625f9bcf4b58bb8343e4316f3601 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:43:01.893836 12117 raft_consensus.cc:3037] T 700f8c61f3c74b908919ee4a7211da5f P 7878625f9bcf4b58bb8343e4316f3601 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:01.895236 12117 raft_consensus.cc:504] T 700f8c61f3c74b908919ee4a7211da5f P 7878625f9bcf4b58bb8343e4316f3601 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a513c734f9449079aae6e92747afdcd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46803 } } peers { permanent_uuid: "7878625f9bcf4b58bb8343e4316f3601" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45733 } } peers { permanent_uuid: "33556bb7a26e4cad99f3656cebaca9e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46465 } }
I20260619 16:43:01.895742 12117 leader_election.cc:288] T 700f8c61f3c74b908919ee4a7211da5f P 7878625f9bcf4b58bb8343e4316f3601 [CANDIDATE]: Term 1 election: Requested vote from peers 6a513c734f9449079aae6e92747afdcd (127.0.98.1:46803), 33556bb7a26e4cad99f3656cebaca9e9 (127.0.98.3:46465)
I20260619 16:43:01.896386 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:01.880981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34367 (local address 127.0.98.3:46465)
0619 16:43:01.881380 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:01.881385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:01.881400 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:43:01.889044 (+ 7644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:01.889049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:01.889115 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:01.889198 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:01.889642 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.890419 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.894115 (+ 3696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.895070 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.895249 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:01.895354 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:01.896119 (+ 765us) server_negotiation.cc:299] Negotiation successful
0619 16:43:01.896252 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296}
I20260619 16:43:01.896703 11774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "700f8c61f3c74b908919ee4a7211da5f" candidate_uuid: "7878625f9bcf4b58bb8343e4316f3601" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a513c734f9449079aae6e92747afdcd"
I20260619 16:43:01.896864 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:01.880353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46465 (local address 127.0.98.2:34367)
0619 16:43:01.880753 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:01.880766 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:01.888040 (+ 7274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:01.889263 (+ 1223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:01.889267 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:01.889299 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:01.889446 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:01.889454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.890608 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.890611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:01.893942 (+ 3331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:01.893959 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.895522 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.895527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:01.895881 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:01.895922 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 16:43:01.896608 (+ 686us) client_negotiation.cc:240] Negotiation successful
0619 16:43:01.896703 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":87,"threads_started":1}
I20260619 16:43:01.896862 11774 raft_consensus.cc:3037] T 700f8c61f3c74b908919ee4a7211da5f P 6a513c734f9449079aae6e92747afdcd [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:01.897533 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "700f8c61f3c74b908919ee4a7211da5f" candidate_uuid: "7878625f9bcf4b58bb8343e4316f3601" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33556bb7a26e4cad99f3656cebaca9e9" is_pre_election: true
I20260619 16:43:01.897876 12039 raft_consensus.cc:2454] T 700f8c61f3c74b908919ee4a7211da5f P 33556bb7a26e4cad99f3656cebaca9e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7878625f9bcf4b58bb8343e4316f3601 in term 0.
I20260619 16:43:01.898281 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "700f8c61f3c74b908919ee4a7211da5f" candidate_uuid: "7878625f9bcf4b58bb8343e4316f3601" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33556bb7a26e4cad99f3656cebaca9e9"
I20260619 16:43:01.898325 11774 raft_consensus.cc:2454] T 700f8c61f3c74b908919ee4a7211da5f P 6a513c734f9449079aae6e92747afdcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7878625f9bcf4b58bb8343e4316f3601 in term 1.
I20260619 16:43:01.898394 12037 raft_consensus.cc:3037] T 700f8c61f3c74b908919ee4a7211da5f P 33556bb7a26e4cad99f3656cebaca9e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:01.898725 11844 leader_election.cc:302] T 700f8c61f3c74b908919ee4a7211da5f P 7878625f9bcf4b58bb8343e4316f3601 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a513c734f9449079aae6e92747afdcd, 7878625f9bcf4b58bb8343e4316f3601; no voters:
I20260619 16:43:01.898929 12117 raft_consensus.cc:2781] T 700f8c61f3c74b908919ee4a7211da5f P 7878625f9bcf4b58bb8343e4316f3601 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:01.899180 12117 raft_consensus.cc:686] T 700f8c61f3c74b908919ee4a7211da5f P 7878625f9bcf4b58bb8343e4316f3601 [term 1 LEADER]: Becoming Leader. State: Replica: 7878625f9bcf4b58bb8343e4316f3601, State: Running, Role: LEADER
I20260619 16:43:01.899539 12117 consensus_queue.cc:237] T 700f8c61f3c74b908919ee4a7211da5f P 7878625f9bcf4b58bb8343e4316f3601 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a513c734f9449079aae6e92747afdcd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46803 } } peers { permanent_uuid: "7878625f9bcf4b58bb8343e4316f3601" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45733 } } peers { permanent_uuid: "33556bb7a26e4cad99f3656cebaca9e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46465 } }
I20260619 16:43:01.899647 12037 raft_consensus.cc:2454] T 700f8c61f3c74b908919ee4a7211da5f P 33556bb7a26e4cad99f3656cebaca9e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7878625f9bcf4b58bb8343e4316f3601 in term 1.
I20260619 16:43:01.904320 11605 catalog_manager.cc:5495] T 700f8c61f3c74b908919ee4a7211da5f P 7878625f9bcf4b58bb8343e4316f3601 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7878625f9bcf4b58bb8343e4316f3601 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7878625f9bcf4b58bb8343e4316f3601" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a513c734f9449079aae6e92747afdcd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7878625f9bcf4b58bb8343e4316f3601" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33556bb7a26e4cad99f3656cebaca9e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46465 } health_report { overall_health: UNKNOWN } } }
W20260619 16:43:01.912040 12091 tablet.cc:2369] T 700f8c61f3c74b908919ee4a7211da5f P 33556bb7a26e4cad99f3656cebaca9e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:43:01.925086 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:01.919571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33950 (local address 127.0.98.2:45733)
0619 16:43:01.919754 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:01.919759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:01.919993 (+ 234us) server_negotiation.cc:407] Connection header received
0619 16:43:01.920162 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:01.920165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:01.920246 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:01.920379 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:43:01.920628 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.921391 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.922714 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:01.923056 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:01.923175 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:01.923651 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:43:01.924294 (+ 643us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:43:01.924434 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:01.924837 (+ 403us) server_negotiation.cc:299] Negotiation successful
0619 16:43:01.924916 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:43:01.934597 12037 raft_consensus.cc:1261] T 700f8c61f3c74b908919ee4a7211da5f P 33556bb7a26e4cad99f3656cebaca9e9 [term 1 FOLLOWER]: Refusing update from remote peer 7878625f9bcf4b58bb8343e4316f3601: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:43:01.934857 11774 raft_consensus.cc:1261] T 700f8c61f3c74b908919ee4a7211da5f P 6a513c734f9449079aae6e92747afdcd [term 1 FOLLOWER]: Refusing update from remote peer 7878625f9bcf4b58bb8343e4316f3601: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:43:01.935539 12117 consensus_queue.cc:1035] T 700f8c61f3c74b908919ee4a7211da5f P 7878625f9bcf4b58bb8343e4316f3601 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33556bb7a26e4cad99f3656cebaca9e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:43:01.936161 12117 consensus_queue.cc:1035] T 700f8c61f3c74b908919ee4a7211da5f P 7878625f9bcf4b58bb8343e4316f3601 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a513c734f9449079aae6e92747afdcd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:43:01.945991 12124 mvcc.cc:205] Tried to move back new op lower bound from 7298610716392026112 to 7298610716263636992. Current Snapshot: MvccSnapshot[applied={T|T < 7298610716392026112}]
I20260619 16:43:01.948343 12125 mvcc.cc:205] Tried to move back new op lower bound from 7298610716392026112 to 7298610716263636992. Current Snapshot: MvccSnapshot[applied={T|T < 7298610716392026112}]
I20260619 16:43:01.949759 12126 mvcc.cc:205] Tried to move back new op lower bound from 7298610716392026112 to 7298610716263636992. Current Snapshot: MvccSnapshot[applied={T|T < 7298610716392026112}]
I20260619 16:43:01.963609 11605 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:43:01.963879 11605 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:43:01.967379 11605 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19419085031646c88b90f96f8f4d5c9c: Sending DeleteTablet for 3 replicas of tablet 700f8c61f3c74b908919ee4a7211da5f
I20260619 16:43:01.968551 11736 tablet_service.cc:1514] Processing DeleteTablet for tablet 700f8c61f3c74b908919ee4a7211da5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59004
I20260619 16:43:01.968616 12009 tablet_service.cc:1514] Processing DeleteTablet for tablet 700f8c61f3c74b908919ee4a7211da5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58432
I20260619 16:43:01.969497 11890 tablet_service.cc:1514] Processing DeleteTablet for tablet 700f8c61f3c74b908919ee4a7211da5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33946
I20260619 16:43:01.970378 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 11687
I20260619 16:43:01.984781 12134 tablet_replica.cc:330] T 700f8c61f3c74b908919ee4a7211da5f P 33556bb7a26e4cad99f3656cebaca9e9: stopping tablet replica
I20260619 16:43:01.985203 12134 raft_consensus.cc:2229] T 700f8c61f3c74b908919ee4a7211da5f P 33556bb7a26e4cad99f3656cebaca9e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:43:01.985615 12134 raft_consensus.cc:2258] T 700f8c61f3c74b908919ee4a7211da5f P 33556bb7a26e4cad99f3656cebaca9e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:43:01.992383 12135 tablet_replica.cc:330] T 700f8c61f3c74b908919ee4a7211da5f P 7878625f9bcf4b58bb8343e4316f3601: stopping tablet replica
W20260619 16:43:01.992318 11591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46803 (error 108)
I20260619 16:43:01.992820 12135 raft_consensus.cc:2229] T 700f8c61f3c74b908919ee4a7211da5f P 7878625f9bcf4b58bb8343e4316f3601 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:43:01.993340 12135 raft_consensus.cc:2258] T 700f8c61f3c74b908919ee4a7211da5f P 7878625f9bcf4b58bb8343e4316f3601 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:43:01.993744 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 11826
I20260619 16:43:01.994747 12134 ts_tablet_manager.cc:1905] T 700f8c61f3c74b908919ee4a7211da5f P 33556bb7a26e4cad99f3656cebaca9e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:43:01.998260 12134 ts_tablet_manager.cc:1918] T 700f8c61f3c74b908919ee4a7211da5f P 33556bb7a26e4cad99f3656cebaca9e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:43:01.998423 12134 log.cc:1196] T 700f8c61f3c74b908919ee4a7211da5f P 33556bb7a26e4cad99f3656cebaca9e9: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887259574980-392-0/minicluster-data/ts-2/wal/wals/700f8c61f3c74b908919ee4a7211da5f
I20260619 16:43:01.998811 12134 ts_tablet_manager.cc:1939] T 700f8c61f3c74b908919ee4a7211da5f P 33556bb7a26e4cad99f3656cebaca9e9: Deleting consensus metadata
I20260619 16:43:02.000504 11592 catalog_manager.cc:4841] TS 33556bb7a26e4cad99f3656cebaca9e9 (127.0.98.3:46465): tablet 700f8c61f3c74b908919ee4a7211da5f (table test-table [id=fc528bbb03264b2da7884b77408efc15]) successfully deleted
I20260619 16:43:02.001750 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:02.001378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46803 (local address 127.0.0.1:59010)
0619 16:43:02.001579 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:02.001657 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260619 16:43:02.002113 11591 catalog_manager.cc:4584] TS 6a513c734f9449079aae6e92747afdcd (127.0.98.1:46803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 700f8c61f3c74b908919ee4a7211da5f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46803: connect: Connection refused (error 111)
I20260619 16:43:02.006698 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 11961
I20260619 16:43:02.008176 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:02.007901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45733 (local address 127.0.0.1:33954)
0619 16:43:02.008011 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:02.008067 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 16:43:02.015026 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 11574
I20260619 16:43:02.023298 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 16:43:02.359354 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 16:43:02.359555 392 mini_ranger.cc:66] Stopping Ranger...
I20260619 16:43:02.705760 392 mini_ranger.cc:68] Stopped Ranger
2026-06-19 16:43:02.706 UTC [11160] LOG: received smart shutdown request
2026-06-19 16:43:02.709 UTC [11160] LOG: background worker "logical replication launcher" (PID 11169) exited with exit code 1
2026-06-19 16:43:02.710 UTC [11164] LOG: shutting down
2026-06-19 16:43:02.919 UTC [11160] LOG: database system is shut down
2026-06-19T16:43:02Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (54793 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:43:03 dist-test-slave-03cd krb5kdc[12145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:43:03 dist-test-slave-03cd krb5kdc[12145](info): set up 2 sockets
Jun 19 16:43:03 dist-test-slave-03cd krb5kdc[12145](info): commencing operation
krb5kdc: starting...
W20260619 16:43:04.132180 392 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:43:04 dist-test-slave-03cd krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:43:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:43:04Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260619 16:43:04.339429 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task8rFIPy/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/postgres -l logfile start
2026-06-19 16:43:07.303 UTC [12175] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 16:43:07.303 UTC [12175] LOG: listening on IPv4 address "127.0.98.20", port 38139
2026-06-19 16:43:07.308 UTC [12175] LOG: listening on Unix socket "/tmp/.s.PGSQL.38139"
2026-06-19 16:43:07.333 UTC [12178] LOG: database system was shut down at 2026-06-19 16:43:05 UTC
2026-06-19 16:43:07.338 UTC [12175] LOG: database system is ready to accept connections
I20260619 16:43:08.328135 392 mini_postgres.cc:96] Postgres bound to 38139
2026-06-19 16:43:08.334 UTC [12186] FATAL: database "slave" does not exist
127.0.98.20:38139 - accepting connections
I20260619 16:43:08.335114 392 mini_ranger.cc:161] Starting Ranger...
I20260619 16:43:08.351989 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 16:43:08.978915 392 mini_ranger.cc:87] Created ranger Postgres database
I20260619 16:43:08.979034 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ranger-admin
2026-06-19 16:43:09,278 [I] DB FLAVOR :POSTGRES
2026-06-19 16:43:09,278 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 16:43:09,279 [I] Checking connection..
2026-06-19 16:43:09,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 16:43:09,532 [I] Checking connection passed.
2026-06-19 16:43:09,532 [I] --------- Verifying version history table ---------
2026-06-19 16:43:09,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:43:09,779 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 16:43:09,779 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:43:10,037 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 16:43:10,037 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 16:43:10,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38139/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 16:43:10.314 UTC [12250] WARNING: there is no transaction in progress
2026-06-19 16:43:10,321 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 16:43:10,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:43:10,570 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 16:43:10,570 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 16:43:10,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 16:43:10,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 16:43:11,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-03cd.c.gcp-upstream.internal','N') ;"
2026-06-19 16:43:11,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 16:43:11,598 [I] Table x_portal_user does not exist in database ranger
2026-06-19 16:43:11,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 16:43:11,852 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 16:43:11,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 16:43:12,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 16:43:12,362 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 16:43:12,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38139/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 16:43:13.520 UTC [12358] WARNING: there is no transaction in progress
2026-06-19 16:43:13.541 UTC [12358] WARNING: there is no transaction in progress
2026-06-19 16:43:13.563 UTC [12358] WARNING: there is no transaction in progress
2026-06-19 16:43:13.581 UTC [12358] WARNING: there is no transaction in progress
2026-06-19 16:43:13.600 UTC [12358] WARNING: there is no transaction in progress
2026-06-19 16:43:13.624 UTC [12358] WARNING: there is no transaction in progress
2026-06-19 16:43:13.688 UTC [12358] WARNING: there is no transaction in progress
2026-06-19 16:43:13.697 UTC [12358] WARNING: there is no transaction in progress
2026-06-19 16:43:13.714 UTC [12358] WARNING: there is no transaction in progress
2026-06-19 16:43:13.727 UTC [12358] WARNING: there is no transaction in progress
2026-06-19 16:43:13.741 UTC [12358] WARNING: there is no transaction in progress
2026-06-19 16:43:13.759 UTC [12358] WARNING: there is no transaction in progress
2026-06-19 16:43:13.771 UTC [12358] WARNING: there is no transaction in progress
2026-06-19 16:43:13.779 UTC [12358] WARNING: there is no transaction in progress
2026-06-19 16:43:13.789 UTC [12358] WARNING: there is no transaction in progress
2026-06-19 16:43:13.798 UTC [12358] WARNING: there is no transaction in progress
2026-06-19 16:43:14,696 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 16:43:14,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 16:43:14,957 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 16:43:14,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 16:43:15,242 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 16:43:15,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 16:43:15,515 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 16:43:15,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 16:43:15,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 16:43:16,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-03cd.c.gcp-upstream.internal';"
2026-06-19 16:43:16,262 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 16:43:16,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 16:43:16,537 [I] DB_PATCHES have already been applied
I20260619 16:43:16.543133 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task8rFIPy/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 16:43:16.543207 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260619 16:43:16.546828 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:36115
Jun 19, 2026 4:43:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:43:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:43:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 4:43:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 4:43:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 4:43:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 4:43:17 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 16:43:17.681948 392 mini_ranger.cc:160] Time spent starting Ranger: real 9.347s user 0.000s sys 0.011s
Jun 19, 2026 4:43:40 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 4:43:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 4:43:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 4:43:40 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 4:43:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 4:43:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 4:43:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 4:43:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 4:43:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 4:43:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 4:43:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 4:43:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 4:43:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 4:43:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 4:43:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 4:43:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 4:43:40 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 4:43:41 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 16:43:41.894484 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 16:43:41 dist-test-slave-03cd krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887421, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 16:43:42.000363 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 16:43:42.011461 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 16:43:43.171772 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 16:43:43.171850 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ranger-kms
I20260619 16:43:43.171864 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:38139
2026-06-19 16:43:43,349 [I] DB FLAVOR :POSTGRES
2026-06-19 16:43:43,349 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 16:43:43,349 [I] Checking connection
2026-06-19 16:43:43,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38139/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 16:43:43,617 [I] connection success
2026-06-19 16:43:43,618 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 16:43:43,618 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 16:43:43,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38139/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 16:43:43,894 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 16:43:43,894 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 16:43:43,894 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 16:43:43,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38139/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 16:43:44,226 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 16:43:44.772753 392 mini_ranger_kms.cc:310] Created kms service
I20260619 16:43:44.983435 392 mini_ranger_kms.cc:326] Created kudu user
I20260619 16:43:45.042059 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 16:43:45 dist-test-slave-03cd krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887421, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:43:45.479090 392 mini_ranger_kms.cc:384] Added ranger policy
I20260619 16:43:45.479249 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task8rFIPy/build/debug/bin/postgresql.jar:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task8rFIPy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 16:43:45.479271 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260619 16:43:45.483093 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12539
I20260619 16:43:45.483201 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:36257
Jun 19, 2026 4:43:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:43:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:43:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 4:43:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 4:43:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 4:43:47 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 16:43:47.767387 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.767s user 0.007s sys 0.155s
I20260619 16:43:47.767518 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 16:43:47.767582 392 mini_ranger_kms.cc:401] 127.0.98.20:36257/kms/v1/keys
Jun 19 16:43:48 dist-test-slave-03cd krb5kdc[12145](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781887428, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 16:43:48 dist-test-slave-03cd krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887428, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 16:43:48 dist-test-slave-03cd krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887428, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 16:43:51 dist-test-slave-03cd krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:43:51.252239 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34179
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34179
--ranger_config_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36257/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:51.386224 12589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:51.386565 12589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:51.386610 12589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:51.391040 12589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:43:51.391137 12589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:51.391158 12589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:51.391180 12589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:43:51.391196 12589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:43:51.397854 12589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:36257/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34179
--ranger_config_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:43:51.399180 12589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:51.401046 12589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:51.410933 12594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:51.410897 12595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:51.411329 12597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:51.411839 12589 server_base.cc:1034] running on GCE node
I20260619 16:43:51.412727 12589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:51.414333 12589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:51.415544 12589 hybrid_clock.cc:648] HybridClock initialized: now 1781887431415510 us; error 51 us; skew 500 ppm
Jun 19 16:43:51 dist-test-slave-03cd krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:51.419080 12589 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:43:51.420332 12589 webserver.cc:460] Webserver started at http://127.0.98.62:40889/ using document root <none> and password file <none>
I20260619 16:43:51.420944 12589 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:51.421005 12589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:51.421258 12589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:43:51 dist-test-slave-03cd krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:43:51.473412 12589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "0bcca7ecb21145f79c740b0641ef7a75"
format_stamp: "Formatted at 2026-06-19 16:43:51 on dist-test-slave-03cd"
server_key: "18f0ce2642e9eda35cac172ca6f9c192"
server_key_iv: "c7390219d8015cdaa4d39796e67f52cd"
server_key_version: "kuduclusterkey@0"
I20260619 16:43:51.474139 12589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "0bcca7ecb21145f79c740b0641ef7a75"
format_stamp: "Formatted at 2026-06-19 16:43:51 on dist-test-slave-03cd"
server_key: "18f0ce2642e9eda35cac172ca6f9c192"
server_key_iv: "c7390219d8015cdaa4d39796e67f52cd"
server_key_version: "kuduclusterkey@0"
I20260619 16:43:51.478492 12589 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:43:51.495609 12604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:51.497236 12589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:43:51.497413 12589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "0bcca7ecb21145f79c740b0641ef7a75"
format_stamp: "Formatted at 2026-06-19 16:43:51 on dist-test-slave-03cd"
server_key: "18f0ce2642e9eda35cac172ca6f9c192"
server_key_iv: "c7390219d8015cdaa4d39796e67f52cd"
server_key_version: "kuduclusterkey@0"
I20260619 16:43:51.497570 12589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:51.523331 12589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:51.526520 12589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:51.526724 12589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:51.534811 12589 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34179
I20260619 16:43:51.534817 12656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34179 every 8 connection(s)
I20260619 16:43:51.535923 12589 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:43:51.538800 12657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:51.542560 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 12589
I20260619 16:43:51.542687 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:43:51.542963 392 external_mini_cluster.cc:1428] Setting key 32dae40c68c3c78976863d068cd3ebb8
I20260619 16:43:51.545248 12657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75: Bootstrap starting.
I20260619 16:43:51.548098 12657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:51.548995 12657 log.cc:824] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:51.551158 12657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75: No bootstrap required, opened a new log
Jun 19 16:43:51 dist-test-slave-03cd krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:43:51.555948 12657 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bcca7ecb21145f79c740b0641ef7a75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34179 } }
I20260619 16:43:51.556244 12657 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:51.556320 12657 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bcca7ecb21145f79c740b0641ef7a75, State: Initialized, Role: FOLLOWER
I20260619 16:43:51.556874 12657 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bcca7ecb21145f79c740b0641ef7a75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34179 } }
I20260619 16:43:51.557051 12657 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:43:51.557155 12657 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:43:51.557269 12657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:51.558550 12657 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bcca7ecb21145f79c740b0641ef7a75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34179 } }
I20260619 16:43:51.558995 12657 leader_election.cc:302] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bcca7ecb21145f79c740b0641ef7a75; no voters:
I20260619 16:43:51.559394 12657 leader_election.cc:288] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:43:51.559553 12662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:51.559868 12662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [term 1 LEADER]: Becoming Leader. State: Replica: 0bcca7ecb21145f79c740b0641ef7a75, State: Running, Role: LEADER
I20260619 16:43:51.560307 12662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bcca7ecb21145f79c740b0641ef7a75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34179 } }
I20260619 16:43:51.560642 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:51.544524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44026 (local address 127.0.98.62:34179)
0619 16:43:51.545025 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:51.545045 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:51.545089 (+ 44us) server_negotiation.cc:407] Connection header received
0619 16:43:51.545958 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:51.545983 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:51.546424 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:51.546817 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:51.547748 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:51.549235 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:51.550469 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:51.551126 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:51.551296 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:51.554464 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:51.554473 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:51.554493 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:51.554532 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:51.556723 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:51.557331 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:51.557337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:51.557346 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:51.557505 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:51.557837 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:51.557841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:51.557842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:51.558453 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:51.558669 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:51.559073 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 16:43:51.559572 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":149,"threads_started":1}
I20260619 16:43:51.560806 12657 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:43:51.561635 12662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bcca7ecb21145f79c740b0641ef7a75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bcca7ecb21145f79c740b0641ef7a75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34179 } } }
I20260619 16:43:51.561743 12662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:51.561931 12662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bcca7ecb21145f79c740b0641ef7a75. Latest consensus state: current_term: 1 leader_uuid: "0bcca7ecb21145f79c740b0641ef7a75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bcca7ecb21145f79c740b0641ef7a75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34179 } } }
I20260619 16:43:51.562026 12662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:51.563179 12657 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 16:43:51.563315 12671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:43:51.569685 12671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:43:51.578828 12671 catalog_manager.cc:1347] Generated new cluster ID: 307f18625a9c41758be46afd1405ee11
I20260619 16:43:51.579036 12671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:43:51.588871 12671 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:43:51.590381 12671 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:43:51.612073 12671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75: Generated new TSK 0
I20260619 16:43:51.613085 12671 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:43:52.335580 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36257/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34179
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:52.462663 12702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:52.462996 12702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:52.463079 12702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:52.467607 12702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:52.467819 12702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:52.467968 12702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:43:52.473908 12702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:36257/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:43:52.475106 12702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:52.476210 12702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:52.486676 12707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:52.487422 12710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:52.487792 12708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:52.487797 12702 server_base.cc:1034] running on GCE node
I20260619 16:43:52.488392 12702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:52.489300 12702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:52.490510 12702 hybrid_clock.cc:648] HybridClock initialized: now 1781887432490484 us; error 48 us; skew 500 ppm
Jun 19 16:43:52 dist-test-slave-03cd krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:52.494416 12702 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:43:52.495682 12702 webserver.cc:460] Webserver started at http://127.0.98.1:42891/ using document root <none> and password file <none>
I20260619 16:43:52.496423 12702 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:52.496512 12702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:52.496760 12702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:43:52 dist-test-slave-03cd krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:43:52.517076 12702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/data/instance:
uuid: "b3899e5f1df84aa6914c3fd46775958f"
format_stamp: "Formatted at 2026-06-19 16:43:52 on dist-test-slave-03cd"
server_key: "04631a0de9c7ee09fbb99bb1a237bd0e"
server_key_iv: "8a6f4bbdd10829211a9ac3ca6fc0080a"
server_key_version: "kuduclusterkey@0"
I20260619 16:43:52.517867 12702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b3899e5f1df84aa6914c3fd46775958f"
format_stamp: "Formatted at 2026-06-19 16:43:52 on dist-test-slave-03cd"
server_key: "04631a0de9c7ee09fbb99bb1a237bd0e"
server_key_iv: "8a6f4bbdd10829211a9ac3ca6fc0080a"
server_key_version: "kuduclusterkey@0"
I20260619 16:43:52.529573 12702 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.006s sys 0.000s
I20260619 16:43:52.551421 12719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:52.553092 12702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:43:52.553222 12702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "b3899e5f1df84aa6914c3fd46775958f"
format_stamp: "Formatted at 2026-06-19 16:43:52 on dist-test-slave-03cd"
server_key: "04631a0de9c7ee09fbb99bb1a237bd0e"
server_key_iv: "8a6f4bbdd10829211a9ac3ca6fc0080a"
server_key_version: "kuduclusterkey@0"
I20260619 16:43:52.553347 12702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:52.578745 12702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:52.582847 12702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:52.583176 12702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:52.584017 12702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:52.585155 12702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:52.585263 12702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:52.585427 12702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:52.585510 12702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:52.600749 12702 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41313
I20260619 16:43:52.602171 12702 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:43:52.606740 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 12702
I20260619 16:43:52.606866 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:43:52.607225 392 external_mini_cluster.cc:1428] Setting key 2e493027c3edc423d193b19b881d9724
I20260619 16:43:52.603929 12832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41313 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Jun 19 16:43:52 dist-test-slave-03cd krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:43:52.633641 12837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:52.607696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41287 (local address 127.0.98.62:34179)
0619 16:43:52.608579 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:52.608588 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:52.609054 (+ 466us) server_negotiation.cc:407] Connection header received
0619 16:43:52.610154 (+ 1100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:52.610158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:52.610214 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:52.610345 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:52.611945 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:52.612908 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:52.620434 (+ 7526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:52.620904 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:52.621001 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:52.628470 (+ 7469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:52.628476 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:52.628479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:52.628519 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:52.630637 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:52.631473 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:52.631477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:52.631479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:52.631586 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:52.632112 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:52.632115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:52.632117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:52.632387 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:52.632485 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:52.633295 (+ 810us) server_negotiation.cc:299] Negotiation successful
0619 16:43:52.633449 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":803,"thread_start_us":116,"threads_started":1}
I20260619 16:43:52.635993 12836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:52.606468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.1:41287)
0619 16:43:52.608902 (+ 2434us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:52.608966 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:52.609925 (+ 959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:52.610536 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:52.610544 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:52.610946 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:52.611742 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:52.611774 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:52.613805 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:52.613812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:52.620177 (+ 6365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:52.620204 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:52.621384 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:52.621390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:52.621652 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:52.623971 (+ 2319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:52.624000 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:52.628166 (+ 4166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:52.630839 (+ 2673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:52.630848 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:52.630872 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:52.631325 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:52.631750 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:52.631753 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:52.631754 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:52.631987 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:52.632559 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:52.632566 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:52.632975 (+ 409us) client_negotiation.cc:769] Sending connection context
0619 16:43:52.634153 (+ 1178us) client_negotiation.cc:240] Negotiation successful
0619 16:43:52.634581 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2169,"thread_start_us":148,"threads_started":1}
I20260619 16:43:52.638129 12833 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34179
I20260619 16:43:52.638446 12833 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:52.639132 12833 heartbeater.cc:511] Master 127.0.98.62:34179 requested a full tablet report, sending...
I20260619 16:43:52.641258 12621 ts_manager.cc:194] Registered new tserver with Master: b3899e5f1df84aa6914c3fd46775958f (127.0.98.1:41313)
I20260619 16:43:52.642868 12621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41287
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:43:52.648300 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36257/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34179
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:43:52 dist-test-slave-03cd krb5kdc[12145](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781887432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 16:43:52.816920 12841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:52.817248 12841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:52.817343 12841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:52.821990 12841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:52.822091 12841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:52.822183 12841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:43:52.830520 12841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:36257/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:43:52.832099 12841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:52.833572 12841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:52.849674 12850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:52.850361 12848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:52.850373 12847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:52.850932 12841 server_base.cc:1034] running on GCE node
I20260619 16:43:52.851543 12841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:52.852474 12841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:52.853718 12841 hybrid_clock.cc:648] HybridClock initialized: now 1781887432853687 us; error 55 us; skew 500 ppm
Jun 19 16:43:52 dist-test-slave-03cd krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:52.857892 12841 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:43:52.859258 12841 webserver.cc:460] Webserver started at http://127.0.98.2:37421/ using document root <none> and password file <none>
I20260619 16:43:52.860092 12841 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:52.860193 12841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:52.860606 12841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:43:52 dist-test-slave-03cd krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:43:52.882109 12841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/data/instance:
uuid: "08b0de52e3f74b048422f4308c73c9ab"
format_stamp: "Formatted at 2026-06-19 16:43:52 on dist-test-slave-03cd"
server_key: "ac8dbfaf9223133ee345b86005c157b1"
server_key_iv: "e2ee5d2a8a397fa7b1da4b12015f130c"
server_key_version: "kuduclusterkey@0"
I20260619 16:43:52.883598 12841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/wal/instance:
uuid: "08b0de52e3f74b048422f4308c73c9ab"
format_stamp: "Formatted at 2026-06-19 16:43:52 on dist-test-slave-03cd"
server_key: "ac8dbfaf9223133ee345b86005c157b1"
server_key_iv: "e2ee5d2a8a397fa7b1da4b12015f130c"
server_key_version: "kuduclusterkey@0"
I20260619 16:43:52.890132 12841 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260619 16:43:52.914889 12857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:52.916764 12841 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260619 16:43:52.916930 12841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "08b0de52e3f74b048422f4308c73c9ab"
format_stamp: "Formatted at 2026-06-19 16:43:52 on dist-test-slave-03cd"
server_key: "ac8dbfaf9223133ee345b86005c157b1"
server_key_iv: "e2ee5d2a8a397fa7b1da4b12015f130c"
server_key_version: "kuduclusterkey@0"
I20260619 16:43:52.917107 12841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:52.979729 12841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:52.983565 12841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:52.983881 12841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:52.984705 12841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:52.986089 12841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:52.986210 12841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:52.986318 12841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:52.986387 12841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:52.998884 12841 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33631
I20260619 16:43:53.000187 12841 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:43:53.000903 12970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33631 every 8 connection(s)
I20260619 16:43:53.004287 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 12841
I20260619 16:43:53.004441 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:43:53.004761 392 external_mini_cluster.cc:1428] Setting key 86a79585b8093914c96f924a2feb7d9b
Jun 19 16:43:53 dist-test-slave-03cd krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 16:43:53.020967 12837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:53.003588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47907 (local address 127.0.98.62:34179)
0619 16:43:53.003750 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:53.003755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:53.005555 (+ 1800us) server_negotiation.cc:407] Connection header received
0619 16:43:53.006844 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:53.006848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:53.006904 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:53.006956 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:53.008755 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:53.009642 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:53.011450 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:53.011827 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:53.011913 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:53.015390 (+ 3477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:53.015396 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:53.015399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:53.015448 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:53.017777 (+ 2329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:53.018730 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:53.018735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:53.018748 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:53.018859 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:53.019364 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:53.019369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:53.019370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:53.019636 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:53.019761 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:53.020592 (+ 831us) server_negotiation.cc:299] Negotiation successful
0619 16:43:53.020763 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:43:53.023108 12973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:53.004117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.2:47907)
0619 16:43:53.005372 (+ 1255us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:53.005446 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:53.006583 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:53.007390 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:53.007399 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:53.007831 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:53.008556 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:53.008578 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:53.009818 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:53.009822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:53.011277 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:53.011290 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:53.012158 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:53.012162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:53.012385 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:53.013265 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:53.013288 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:53.015132 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:53.018086 (+ 2954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:53.018095 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:53.018118 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:53.018581 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:53.018999 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:53.019003 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:53.019005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:53.019233 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:53.019857 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:53.019865 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:53.020286 (+ 421us) client_negotiation.cc:769] Sending connection context
0619 16:43:53.021456 (+ 1170us) client_negotiation.cc:240] Negotiation successful
0619 16:43:53.021870 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1019,"thread_start_us":168,"threads_started":1}
I20260619 16:43:53.024969 12971 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34179
I20260619 16:43:53.025249 12971 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:53.025878 12971 heartbeater.cc:511] Master 127.0.98.62:34179 requested a full tablet report, sending...
I20260619 16:43:53.027230 12621 ts_manager.cc:194] Registered new tserver with Master: 08b0de52e3f74b048422f4308c73c9ab (127.0.98.2:33631)
I20260619 16:43:53.028054 12621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:43:53.048364 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36257/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34179
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:43:53 dist-test-slave-03cd krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260619 16:43:53.228297 12976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:53.228814 12976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:53.228931 12976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:53.236243 12976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:53.236439 12976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:53.236594 12976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:43:53.246179 12976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:36257/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:43:53.248032 12976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:53.249557 12976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:53.265908 12984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:53.267961 12976 server_base.cc:1034] running on GCE node
W20260619 16:43:53.268827 12982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:53.271863 12981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:53.272667 12976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:53.273641 12976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:53.276935 12976 hybrid_clock.cc:648] HybridClock initialized: now 1781887433276851 us; error 100 us; skew 500 ppm
Jun 19 16:43:53 dist-test-slave-03cd krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:53.281608 12976 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:43:53.282992 12976 webserver.cc:460] Webserver started at http://127.0.98.3:44925/ using document root <none> and password file <none>
I20260619 16:43:53.284413 12976 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:53.284583 12976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:53.284899 12976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:43:53 dist-test-slave-03cd krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:43:53.317333 12976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/data/instance:
uuid: "6b8739a210f9486290e6d232c904acde"
format_stamp: "Formatted at 2026-06-19 16:43:53 on dist-test-slave-03cd"
server_key: "bddd14b04471247b8aa5de2cbde90d85"
server_key_iv: "ba1728fadae53bad25877ad279386f82"
server_key_version: "kuduclusterkey@0"
I20260619 16:43:53.318619 12976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6b8739a210f9486290e6d232c904acde"
format_stamp: "Formatted at 2026-06-19 16:43:53 on dist-test-slave-03cd"
server_key: "bddd14b04471247b8aa5de2cbde90d85"
server_key_iv: "ba1728fadae53bad25877ad279386f82"
server_key_version: "kuduclusterkey@0"
I20260619 16:43:53.331755 12976 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.005s sys 0.000s
I20260619 16:43:53.368532 12991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:53.370467 12976 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:43:53.370645 12976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "6b8739a210f9486290e6d232c904acde"
format_stamp: "Formatted at 2026-06-19 16:43:53 on dist-test-slave-03cd"
server_key: "bddd14b04471247b8aa5de2cbde90d85"
server_key_iv: "ba1728fadae53bad25877ad279386f82"
server_key_version: "kuduclusterkey@0"
I20260619 16:43:53.370828 12976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:53.389981 12976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:53.393951 12976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:53.394181 12976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:53.394963 12976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:53.396097 12976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:53.396168 12976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:53.396237 12976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:53.396271 12976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:53.408519 12976 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46195
I20260619 16:43:53.408569 13104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46195 every 8 connection(s)
I20260619 16:43:53.409981 12976 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:43:53.419708 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 12976
I20260619 16:43:53.419842 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:43:53.420174 392 external_mini_cluster.cc:1428] Setting key 97f73e9a6e5b0e51a08ff40697c327af
Jun 19 16:43:53 dist-test-slave-03cd krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:43:53.432147 12837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:53.414236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35229 (local address 127.0.98.62:34179)
0619 16:43:53.414385 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:53.414391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:53.416055 (+ 1664us) server_negotiation.cc:407] Connection header received
0619 16:43:53.417924 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:53.417931 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:53.418016 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:53.418155 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:53.419887 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:53.420753 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:53.423716 (+ 2963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:53.424085 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:53.424203 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:53.427489 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:53.427495 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:53.427498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:53.427532 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:53.429384 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:53.430110 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:53.430115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:53.430117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:53.430221 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:53.430702 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:53.430706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:53.430708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:53.430980 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:53.431094 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:53.431823 (+ 729us) server_negotiation.cc:299] Negotiation successful
0619 16:43:53.431965 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:43:53.433606 13107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:53.413095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.3:35229)
0619 16:43:53.415822 (+ 2727us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:53.415898 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:53.417597 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:53.418436 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:53.418447 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:53.418842 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:53.419611 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:53.419633 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:53.420933 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:53.420937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:53.423429 (+ 2492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:53.423445 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:53.424299 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:53.424304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:53.424537 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:53.425322 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:53.425345 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:53.427225 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:53.429536 (+ 2311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:53.429544 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:53.429566 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:53.429975 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:53.430351 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:53.430354 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:53.430357 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:53.430574 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:53.431103 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:53.431111 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:53.431506 (+ 395us) client_negotiation.cc:769] Sending connection context
0619 16:43:53.431780 (+ 274us) client_negotiation.cc:240] Negotiation successful
0619 16:43:53.432160 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2413,"thread_start_us":197,"threads_started":1}
I20260619 16:43:53.435076 13105 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34179
I20260619 16:43:53.435375 13105 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:53.436043 13105 heartbeater.cc:511] Master 127.0.98.62:34179 requested a full tablet report, sending...
I20260619 16:43:53.437207 12621 ts_manager.cc:194] Registered new tserver with Master: 6b8739a210f9486290e6d232c904acde (127.0.98.3:46195)
I20260619 16:43:53.437973 12621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35229
I20260619 16:43:53.447917 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:43:53.448770 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 12702
I20260619 16:43:53.455972 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 12841
I20260619 16:43:53.463105 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 12976
I20260619 16:43:53.470057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 12589
I20260619 16:43:53.479132 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34179
--webserver_interface=127.0.98.62
--webserver_port=40889
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34179
--ranger_config_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36257/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36257/kms with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:53.592969 13108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:53.593271 13108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:53.593312 13108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:53.597460 13108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:43:53.597530 13108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:53.597549 13108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:53.597565 13108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:43:53.597618 13108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:43:53.603404 13108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36257/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34179
--ranger_config_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:43:53.604650 13108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:53.605711 13108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:53.617762 13114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:53.617762 13113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:53.618331 13116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:53.618423 13108 server_base.cc:1034] running on GCE node
I20260619 16:43:53.619083 13108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:53.620345 13108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:53.621587 13108 hybrid_clock.cc:648] HybridClock initialized: now 1781887433621578 us; error 34 us; skew 500 ppm
Jun 19 16:43:53 dist-test-slave-03cd krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:53.624840 13108 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:43:53.625887 13108 webserver.cc:460] Webserver started at http://127.0.98.62:40889/ using document root <none> and password file <none>
I20260619 16:43:53.626505 13108 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:53.626609 13108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 16:43:53 dist-test-slave-03cd krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:43:53.674870 13108 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:43:53.677747 13124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:53.679111 13108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:43:53.679281 13108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "0bcca7ecb21145f79c740b0641ef7a75"
format_stamp: "Formatted at 2026-06-19 16:43:51 on dist-test-slave-03cd"
server_key: "18f0ce2642e9eda35cac172ca6f9c192"
server_key_iv: "c7390219d8015cdaa4d39796e67f52cd"
server_key_version: "kuduclusterkey@0"
I20260619 16:43:53.679939 13108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:53.694622 13108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:53.697794 13108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:53.698310 13108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:53.705607 13108 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34179
I20260619 16:43:53.705659 13176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34179 every 8 connection(s)
I20260619 16:43:53.706696 13108 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:43:53.707266 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 13108
I20260619 16:43:53.707794 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36257/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41313
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34179
--webserver_port=42891
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36257/kms with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:43:53.711555 13177 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:43:53.714447 13177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75: Bootstrap starting.
I20260619 16:43:53.737262 13177 log.cc:824] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:53.745709 13177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:43:53.746186 13177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75: Bootstrap complete.
I20260619 16:43:53.751717 13177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bcca7ecb21145f79c740b0641ef7a75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34179 } }
I20260619 16:43:53.752290 13177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bcca7ecb21145f79c740b0641ef7a75, State: Initialized, Role: FOLLOWER
I20260619 16:43:53.752890 13177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bcca7ecb21145f79c740b0641ef7a75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34179 } }
I20260619 16:43:53.753046 13177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:43:53.753113 13177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:43:53.753206 13177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:43:53.755851 13177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bcca7ecb21145f79c740b0641ef7a75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34179 } }
I20260619 16:43:53.756237 13177 leader_election.cc:302] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bcca7ecb21145f79c740b0641ef7a75; no voters:
I20260619 16:43:53.756578 13177 leader_election.cc:288] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 16:43:53.756799 13181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:43:53.757069 13181 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [term 2 LEADER]: Becoming Leader. State: Replica: 0bcca7ecb21145f79c740b0641ef7a75, State: Running, Role: LEADER
I20260619 16:43:53.757445 13181 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bcca7ecb21145f79c740b0641ef7a75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34179 } }
I20260619 16:43:53.757967 13177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:43:53.758890 13183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bcca7ecb21145f79c740b0641ef7a75. Latest consensus state: current_term: 2 leader_uuid: "0bcca7ecb21145f79c740b0641ef7a75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bcca7ecb21145f79c740b0641ef7a75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34179 } } }
I20260619 16:43:53.759021 13183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:53.759305 13182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0bcca7ecb21145f79c740b0641ef7a75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bcca7ecb21145f79c740b0641ef7a75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34179 } } }
I20260619 16:43:53.759380 13182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75 [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:53.759398 13190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:43:53.762679 13190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:43:53.763271 13177 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 16:43:53.763377 13190 catalog_manager.cc:1259] Loaded cluster ID: 307f18625a9c41758be46afd1405ee11
I20260619 16:43:53.763427 13190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:43:53.765803 13190 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:43:53.766456 13190 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75: Loaded TSK: 0
I20260619 16:43:53.767724 13190 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:43:53.867923 13179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:53.868396 13179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:53.868485 13179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:53.874856 13179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:53.875008 13179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:53.875137 13179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:43:53.883575 13179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36257/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42891
--tserver_master_addrs=127.0.98.62:34179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:43:53.885167 13179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:53.886497 13179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:53.899313 13217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:53.899696 13214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:53.900241 13179 server_base.cc:1034] running on GCE node
W20260619 16:43:53.901494 13215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:53.902037 13179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:53.902827 13179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:53.904039 13179 hybrid_clock.cc:648] HybridClock initialized: now 1781887433904007 us; error 55 us; skew 500 ppm
Jun 19 16:43:53 dist-test-slave-03cd krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:53.908392 13179 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:43:53.909737 13179 webserver.cc:460] Webserver started at http://127.0.98.1:42891/ using document root <none> and password file <none>
I20260619 16:43:53.910499 13179 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:53.910648 13179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 16:43:53 dist-test-slave-03cd krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:43:53.937554 13179 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:43:53.940353 13225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:53.941850 13179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:43:53.942059 13179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "b3899e5f1df84aa6914c3fd46775958f"
format_stamp: "Formatted at 2026-06-19 16:43:52 on dist-test-slave-03cd"
server_key: "04631a0de9c7ee09fbb99bb1a237bd0e"
server_key_iv: "8a6f4bbdd10829211a9ac3ca6fc0080a"
server_key_version: "kuduclusterkey@0"
I20260619 16:43:53.942821 13179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:53.964804 13179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:53.969182 13179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:53.969560 13179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:53.970383 13179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:53.971624 13179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:53.971783 13179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:53.971963 13179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:53.972066 13179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:53.983685 13179 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41313
I20260619 16:43:53.985162 13179 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:43:53.987738 13338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41313 every 8 connection(s)
I20260619 16:43:53.988948 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 13179
I20260619 16:43:53.989450 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36257/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33631
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34179
--webserver_port=37421
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36257/kms with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:43:54 dist-test-slave-03cd krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:43:54.014794 13342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:53.991176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.1:40707)
0619 16:43:53.993659 (+ 2483us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:53.993719 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:53.994852 (+ 1133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:53.997057 (+ 2205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:53.997067 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:53.997518 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:53.998313 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:53.998342 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:54.000715 (+ 2373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:54.000720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:54.002610 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:54.002632 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:54.003800 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:54.003804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:54.004002 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:54.004758 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:54.004781 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:54.007292 (+ 2511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:54.010334 (+ 3042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:54.010342 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:54.010360 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:54.010793 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:54.011295 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:54.011298 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:54.011300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:54.011514 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:54.012504 (+ 990us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:54.012511 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:54.012892 (+ 381us) client_negotiation.cc:769] Sending connection context
0619 16:43:54.013138 (+ 246us) client_negotiation.cc:240] Negotiation successful
0619 16:43:54.013577 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2237,"thread_start_us":169,"threads_started":1}
I20260619 16:43:54.016871 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:53.991682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40707 (local address 127.0.98.62:34179)
0619 16:43:53.993321 (+ 1639us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:53.993346 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:53.993882 (+ 536us) server_negotiation.cc:407] Connection header received
0619 16:43:53.995435 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:53.995456 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:53.995938 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:53.996294 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:53.998586 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:54.000480 (+ 1894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:54.002828 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:54.003561 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:54.003817 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:54.007599 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:54.007607 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:54.007628 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:54.007713 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:54.010151 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:54.010967 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:54.010975 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:54.010982 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:54.011162 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:54.011642 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:54.011646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:54.011648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:54.012291 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:54.012427 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:54.015266 (+ 2839us) server_negotiation.cc:299] Negotiation successful
0619 16:43:54.015684 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1452,"thread_start_us":111,"threads_started":1}
I20260619 16:43:54.020694 13339 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34179
I20260619 16:43:54.020983 13339 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:54.021610 13339 heartbeater.cc:511] Master 127.0.98.62:34179 requested a full tablet report, sending...
W20260619 16:43:54.024356 13339 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34179 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 16:43:54.024528 13339 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:54.024780 13339 heartbeater.cc:511] Master 127.0.98.62:34179 requested a full tablet report, sending...
I20260619 16:43:54.025873 13339 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:54.026098 13339 heartbeater.cc:511] Master 127.0.98.62:34179 requested a full tablet report, sending...
W20260619 16:43:54.027064 13339 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:43:54.160590 13341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:54.160889 13341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:54.160934 13341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:54.167104 13341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:54.167215 13341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:54.167320 13341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:43:54.174106 13341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36257/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33631
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37421
--tserver_master_addrs=127.0.98.62:34179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:43:54.175572 13341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:54.176936 13341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:54.189139 13349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:54.189137 13348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:54.190040 13351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:54.190172 13341 server_base.cc:1034] running on GCE node
I20260619 16:43:54.190709 13341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:54.191505 13341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:54.192703 13341 hybrid_clock.cc:648] HybridClock initialized: now 1781887434192668 us; error 74 us; skew 500 ppm
Jun 19 16:43:54 dist-test-slave-03cd krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:54.196488 13341 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:43:54.197985 13341 webserver.cc:460] Webserver started at http://127.0.98.2:37421/ using document root <none> and password file <none>
I20260619 16:43:54.198918 13341 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:54.199106 13341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 16:43:54 dist-test-slave-03cd krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:43:54.233402 13341 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.000s
I20260619 16:43:54.236567 13359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:54.238068 13341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:43:54.238291 13341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "08b0de52e3f74b048422f4308c73c9ab"
format_stamp: "Formatted at 2026-06-19 16:43:52 on dist-test-slave-03cd"
server_key: "ac8dbfaf9223133ee345b86005c157b1"
server_key_iv: "e2ee5d2a8a397fa7b1da4b12015f130c"
server_key_version: "kuduclusterkey@0"
I20260619 16:43:54.238936 13341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:54.295065 13341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:54.298650 13341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:54.298960 13341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:54.299614 13341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:54.300704 13341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:54.300755 13341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:54.300804 13341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:54.300858 13341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:54.311796 13473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33631 every 8 connection(s)
I20260619 16:43:54.312697 13341 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33631
I20260619 16:43:54.313854 13341 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:43:54.323275 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 13341
I20260619 16:43:54.323882 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36257/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46195
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34179
--webserver_port=44925
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36257/kms with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:43:54 dist-test-slave-03cd krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:43:54.330197 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:54.315834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40453 (local address 127.0.98.62:34179)
0619 16:43:54.315998 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:54.316003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:54.316917 (+ 914us) server_negotiation.cc:407] Connection header received
0619 16:43:54.318077 (+ 1160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:54.318081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:54.318142 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:54.318255 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:54.319389 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:54.320257 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:54.321510 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:54.321877 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:54.321986 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:54.325324 (+ 3338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:54.325329 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:54.325332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:54.325366 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:54.327107 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:54.327968 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:54.327972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:54.327974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:54.328074 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:54.328612 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:54.328615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:54.328617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:54.328890 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:54.329001 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:54.329850 (+ 849us) server_negotiation.cc:299] Negotiation successful
0619 16:43:54.330017 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:43:54.331482 13476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:54.316284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.2:40453)
0619 16:43:54.316763 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:54.316815 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:54.317819 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:54.318435 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:54.318442 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:54.318712 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:54.319209 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:54.319223 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:54.320421 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:54.320424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:54.321371 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:54.321382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:54.322056 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:54.322059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:54.322218 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:54.323010 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:54.323037 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:54.324959 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:54.327263 (+ 2304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:54.327274 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:54.327300 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:54.327816 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:54.328205 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:54.328212 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:54.328217 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:54.328469 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:54.329042 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:54.329055 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:54.329521 (+ 466us) client_negotiation.cc:769] Sending connection context
0619 16:43:54.329778 (+ 257us) client_negotiation.cc:240] Negotiation successful
0619 16:43:54.330215 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":149,"threads_started":1}
I20260619 16:43:54.333222 13474 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34179
I20260619 16:43:54.333518 13474 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:54.334177 13474 heartbeater.cc:511] Master 127.0.98.62:34179 requested a full tablet report, sending...
W20260619 16:43:54.336180 13474 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34179 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 16:43:54.336328 13474 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:54.336550 13474 heartbeater.cc:511] Master 127.0.98.62:34179 requested a full tablet report, sending...
I20260619 16:43:54.337703 13474 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:54.337917 13474 heartbeater.cc:511] Master 127.0.98.62:34179 requested a full tablet report, sending...
W20260619 16:43:54.339079 13474 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:43:54.475193 13477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:54.475575 13477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:54.475647 13477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:54.482013 13477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:54.482122 13477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:54.482227 13477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:43:54.488520 13477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36257/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44925
--tserver_master_addrs=127.0.98.62:34179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:43:54.489782 13477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:54.491186 13477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:54.503789 13485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:54.503922 13482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:54.504276 13477 server_base.cc:1034] running on GCE node
W20260619 16:43:54.503783 13483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:54.504796 13477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:54.505676 13477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:54.506877 13477 hybrid_clock.cc:648] HybridClock initialized: now 1781887434506840 us; error 61 us; skew 500 ppm
Jun 19 16:43:54 dist-test-slave-03cd krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:54.510322 13477 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:43:54.511405 13477 webserver.cc:460] Webserver started at http://127.0.98.3:44925/ using document root <none> and password file <none>
I20260619 16:43:54.512075 13477 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:54.512183 13477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 16:43:54 dist-test-slave-03cd krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:43:54.536263 13477 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:43:54.539567 13493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:54.541307 13477 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 16:43:54.541476 13477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "6b8739a210f9486290e6d232c904acde"
format_stamp: "Formatted at 2026-06-19 16:43:53 on dist-test-slave-03cd"
server_key: "bddd14b04471247b8aa5de2cbde90d85"
server_key_iv: "ba1728fadae53bad25877ad279386f82"
server_key_version: "kuduclusterkey@0"
I20260619 16:43:54.542295 13477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:54.565654 13477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:54.569670 13477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:54.569912 13477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:54.570590 13477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:54.571563 13477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:54.571651 13477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:54.571748 13477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:54.571810 13477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:54.583170 13477 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46195
I20260619 16:43:54.583390 13613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46195 every 8 connection(s)
I20260619 16:43:54.584471 13477 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:43:54.586236 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 13477
Jun 19 16:43:54 dist-test-slave-03cd krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:43:54.598340 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:54.588067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55640 (local address 127.0.98.62:34179)
0619 16:43:54.588381 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:54.588392 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:54.588652 (+ 260us) server_negotiation.cc:407] Connection header received
0619 16:43:54.588826 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:54.588830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:54.588885 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:54.588988 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:54.589869 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:54.590629 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:54.591879 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:54.592099 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:54.592219 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:54.593831 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:54.593836 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:54.593839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:54.593903 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:54.595817 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:54.596309 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:54.596313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:54.596315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:54.596418 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:54.596734 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:54.596739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:54.596741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:54.596992 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:54.597207 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:54.597986 (+ 779us) server_negotiation.cc:299] Negotiation successful
0619 16:43:54.598135 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"spinlock_wait_cycles":19072,"thread_start_us":131,"threads_started":1}
I20260619 16:43:54.601253 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:54.586615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53463 (local address 127.0.98.62:34179)
0619 16:43:54.586763 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:54.586768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:54.587842 (+ 1074us) server_negotiation.cc:407] Connection header received
0619 16:43:54.589243 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:54.589248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:54.589298 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:54.589391 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:54.590856 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:54.591635 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:54.593344 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:54.593696 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:54.593793 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:54.596796 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:54.596800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:54.596802 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:54.596835 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:54.598559 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:54.599480 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:54.599484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:54.599485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:54.599580 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:54.600039 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:54.600043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:54.600045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:54.600303 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:54.600397 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:54.600960 (+ 563us) server_negotiation.cc:299] Negotiation successful
0619 16:43:54.601086 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44,"spinlock_wait_cycles":11392}
I20260619 16:43:54.602108 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:54.587094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.3:53463)
0619 16:43:54.587630 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:54.587713 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:54.589003 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:54.589616 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:54.589626 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:54.589991 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:54.590710 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:54.590730 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:54.591781 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:54.591785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:54.593199 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:54.593211 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:54.594017 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:54.594020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:54.594210 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:54.594781 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:54.594798 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:54.596585 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:54.598693 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:54.598699 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:54.598720 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:54.599066 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:54.599756 (+ 690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:54.599758 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:54.599759 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:54.599934 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:54.600397 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:54.600406 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:54.600706 (+ 300us) client_negotiation.cc:769] Sending connection context
0619 16:43:54.600906 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 16:43:54.601266 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":123,"threads_started":1}
I20260619 16:43:54.603264 13614 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34179
I20260619 16:43:54.603549 13614 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:54.604218 13614 heartbeater.cc:511] Master 127.0.98.62:34179 requested a full tablet report, sending...
I20260619 16:43:54.605816 13141 ts_manager.cc:194] Registered new tserver with Master: 6b8739a210f9486290e6d232c904acde (127.0.98.3:46195)
I20260619 16:43:54.607476 13141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53463
Jun 19 16:43:54 dist-test-slave-03cd krb5kdc[12145](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781887434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:55.027349 13339 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:55.027711 13339 heartbeater.cc:511] Master 127.0.98.62:34179 requested a full tablet report, sending...
I20260619 16:43:55.028434 13141 ts_manager.cc:194] Registered new tserver with Master: b3899e5f1df84aa6914c3fd46775958f (127.0.98.1:41313)
I20260619 16:43:55.029057 13141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40707
I20260619 16:43:55.339298 13474 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:55.339627 13474 heartbeater.cc:511] Master 127.0.98.62:34179 requested a full tablet report, sending...
I20260619 16:43:55.340464 13141 ts_manager.cc:194] Registered new tserver with Master: 08b0de52e3f74b048422f4308c73c9ab (127.0.98.2:33631)
Jun 19 16:43:55 dist-test-slave-03cd krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:43:55.341205 13141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40453
I20260619 16:43:55.353767 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:43:55.372763 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:55.358282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55654 (local address 127.0.98.62:34179)
0619 16:43:55.362603 (+ 4321us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:55.362611 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:55.362918 (+ 307us) server_negotiation.cc:407] Connection header received
0619 16:43:55.363131 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:55.363135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:55.363188 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:55.363252 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:55.364263 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.365081 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.366227 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.366552 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.366625 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:55.368568 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:55.368572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:55.368575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:55.368607 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:55.370396 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:55.370948 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:55.370952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:55.370954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:55.371051 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:55.371349 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:55.371352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:55.371354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:55.371577 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:55.371724 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:55.372447 (+ 723us) server_negotiation.cc:299] Negotiation successful
0619 16:43:55.372584 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4196,"thread_start_us":95,"threads_started":1}
I20260619 16:43:55.377614 13141 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:43:55.382910 13141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:43:55.412907 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:55.399832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33631 (local address 127.0.0.1:43584)
0619 16:43:55.402679 (+ 2847us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:55.402703 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:55.402795 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:55.403613 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:55.403620 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:55.403645 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:55.404713 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:55.404725 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.406245 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.406249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:55.409289 (+ 3040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:55.409300 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.412265 (+ 2965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.412269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:55.412410 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:55.412458 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 16:43:55.412581 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 16:43:55.412734 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2790,"thread_start_us":91,"threads_started":1}
I20260619 16:43:55.413753 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:55.400463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43584 (local address 127.0.98.2:33631)
0619 16:43:55.402198 (+ 1735us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:55.402207 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:55.402895 (+ 688us) server_negotiation.cc:407] Connection header received
0619 16:43:55.402986 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:55.402992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:55.403233 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:55.403397 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:55.404863 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.406093 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.409435 (+ 3342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.410325 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.410429 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:55.410602 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:55.413473 (+ 2871us) server_negotiation.cc:299] Negotiation successful
0619 16:43:55.413603 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1621,"thread_start_us":110,"threads_started":1}
I20260619 16:43:55.416268 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:55.397822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41313 (local address 127.0.0.1:51900)
0619 16:43:55.400525 (+ 2703us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:55.400584 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:55.400721 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:55.408674 (+ 7953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:55.408679 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:55.408701 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:55.408797 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:55.408803 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.410155 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.410158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:55.411235 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:55.411244 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.415856 (+ 4612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.415860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:55.415961 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:55.415989 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 16:43:55.416040 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 16:43:55.416119 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2512,"thread_start_us":103,"threads_started":1}
I20260619 16:43:55.417322 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:55.398640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51900 (local address 127.0.98.1:41313)
0619 16:43:55.407920 (+ 9280us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:55.407930 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:55.407962 (+ 32us) server_negotiation.cc:407] Connection header received
0619 16:43:55.408050 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:55.408056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:55.408340 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:55.408511 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:55.408918 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.410019 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.414725 (+ 4706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.415710 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.416794 (+ 1084us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:55.416984 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:55.417040 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 16:43:55.417144 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9143,"thread_start_us":96,"threads_started":1}
I20260619 16:43:55.419082 13408 tablet_service.cc:1467] Processing CreateTablet for tablet 31a3e455b68541ddbf94686ec4d469fb (DEFAULT_TABLE table=test-table [id=096374bfc4fa4d5c85a5e60ad099abe2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:55.420367 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:55.397965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46195 (local address 127.0.0.1:34702)
0619 16:43:55.402345 (+ 4380us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:55.402360 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:55.402454 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:55.410707 (+ 8253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:55.410711 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:55.410731 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:55.410824 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:55.410829 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.417565 (+ 6736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.417569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:55.418712 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:55.418725 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.419936 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.419940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:55.420034 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:55.420061 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 16:43:55.420112 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 16:43:55.420216 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4316,"mutex_wait_us":1,"thread_start_us":87,"threads_started":1}
I20260619 16:43:55.420454 13408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31a3e455b68541ddbf94686ec4d469fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:55.420174 13254 tablet_service.cc:1467] Processing CreateTablet for tablet 31a3e455b68541ddbf94686ec4d469fb (DEFAULT_TABLE table=test-table [id=096374bfc4fa4d5c85a5e60ad099abe2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:55.421156 13254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31a3e455b68541ddbf94686ec4d469fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:55.421383 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:55.398145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34702 (local address 127.0.98.3:46195)
0619 16:43:55.408333 (+ 10188us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:55.408339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:55.408361 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:43:55.408434 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:55.408441 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:55.408600 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:55.410989 (+ 2389us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:55.411126 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.415257 (+ 4131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.418897 (+ 3640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.419760 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.420919 (+ 1159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:55.421080 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:55.421136 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 16:43:55.421245 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10078,"thread_start_us":126,"threads_started":1}
I20260619 16:43:55.424333 13530 tablet_service.cc:1467] Processing CreateTablet for tablet 31a3e455b68541ddbf94686ec4d469fb (DEFAULT_TABLE table=test-table [id=096374bfc4fa4d5c85a5e60ad099abe2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:55.425436 13530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31a3e455b68541ddbf94686ec4d469fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:55.436272 13641 tablet_bootstrap.cc:492] T 31a3e455b68541ddbf94686ec4d469fb P 08b0de52e3f74b048422f4308c73c9ab: Bootstrap starting.
I20260619 16:43:55.438990 13641 tablet_bootstrap.cc:654] T 31a3e455b68541ddbf94686ec4d469fb P 08b0de52e3f74b048422f4308c73c9ab: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:55.440025 13641 log.cc:824] T 31a3e455b68541ddbf94686ec4d469fb P 08b0de52e3f74b048422f4308c73c9ab: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:55.445724 13641 tablet_bootstrap.cc:492] T 31a3e455b68541ddbf94686ec4d469fb P 08b0de52e3f74b048422f4308c73c9ab: No bootstrap required, opened a new log
I20260619 16:43:55.446061 13641 ts_tablet_manager.cc:1397] T 31a3e455b68541ddbf94686ec4d469fb P 08b0de52e3f74b048422f4308c73c9ab: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 16:43:55.450686 13641 raft_consensus.cc:348] T 31a3e455b68541ddbf94686ec4d469fb P 08b0de52e3f74b048422f4308c73c9ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3899e5f1df84aa6914c3fd46775958f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41313 } } peers { permanent_uuid: "6b8739a210f9486290e6d232c904acde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46195 } } peers { permanent_uuid: "08b0de52e3f74b048422f4308c73c9ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33631 } }
I20260619 16:43:55.451120 13641 raft_consensus.cc:374] T 31a3e455b68541ddbf94686ec4d469fb P 08b0de52e3f74b048422f4308c73c9ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:55.451208 13641 raft_consensus.cc:729] T 31a3e455b68541ddbf94686ec4d469fb P 08b0de52e3f74b048422f4308c73c9ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08b0de52e3f74b048422f4308c73c9ab, State: Initialized, Role: FOLLOWER
I20260619 16:43:55.451777 13641 consensus_queue.cc:260] T 31a3e455b68541ddbf94686ec4d469fb P 08b0de52e3f74b048422f4308c73c9ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3899e5f1df84aa6914c3fd46775958f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41313 } } peers { permanent_uuid: "6b8739a210f9486290e6d232c904acde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46195 } } peers { permanent_uuid: "08b0de52e3f74b048422f4308c73c9ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33631 } }
I20260619 16:43:55.452986 13641 ts_tablet_manager.cc:1428] T 31a3e455b68541ddbf94686ec4d469fb P 08b0de52e3f74b048422f4308c73c9ab: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 16:43:55.453423 13474 heartbeater.cc:503] Master 127.0.98.62:34179 was elected leader, sending a full tablet report...
I20260619 16:43:55.453500 13644 tablet_bootstrap.cc:492] T 31a3e455b68541ddbf94686ec4d469fb P 6b8739a210f9486290e6d232c904acde: Bootstrap starting.
I20260619 16:43:55.455346 13642 tablet_bootstrap.cc:492] T 31a3e455b68541ddbf94686ec4d469fb P b3899e5f1df84aa6914c3fd46775958f: Bootstrap starting.
I20260619 16:43:55.461218 13644 tablet_bootstrap.cc:654] T 31a3e455b68541ddbf94686ec4d469fb P 6b8739a210f9486290e6d232c904acde: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:55.461387 13642 tablet_bootstrap.cc:654] T 31a3e455b68541ddbf94686ec4d469fb P b3899e5f1df84aa6914c3fd46775958f: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:55.462220 13644 log.cc:824] T 31a3e455b68541ddbf94686ec4d469fb P 6b8739a210f9486290e6d232c904acde: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:55.463101 13642 log.cc:824] T 31a3e455b68541ddbf94686ec4d469fb P b3899e5f1df84aa6914c3fd46775958f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:55.465590 13642 tablet_bootstrap.cc:492] T 31a3e455b68541ddbf94686ec4d469fb P b3899e5f1df84aa6914c3fd46775958f: No bootstrap required, opened a new log
I20260619 16:43:55.465818 13642 ts_tablet_manager.cc:1397] T 31a3e455b68541ddbf94686ec4d469fb P b3899e5f1df84aa6914c3fd46775958f: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260619 16:43:55.470563 13642 raft_consensus.cc:348] T 31a3e455b68541ddbf94686ec4d469fb P b3899e5f1df84aa6914c3fd46775958f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3899e5f1df84aa6914c3fd46775958f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41313 } } peers { permanent_uuid: "6b8739a210f9486290e6d232c904acde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46195 } } peers { permanent_uuid: "08b0de52e3f74b048422f4308c73c9ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33631 } }
I20260619 16:43:55.470973 13642 raft_consensus.cc:374] T 31a3e455b68541ddbf94686ec4d469fb P b3899e5f1df84aa6914c3fd46775958f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:55.471062 13642 raft_consensus.cc:729] T 31a3e455b68541ddbf94686ec4d469fb P b3899e5f1df84aa6914c3fd46775958f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3899e5f1df84aa6914c3fd46775958f, State: Initialized, Role: FOLLOWER
I20260619 16:43:55.471686 13642 consensus_queue.cc:260] T 31a3e455b68541ddbf94686ec4d469fb P b3899e5f1df84aa6914c3fd46775958f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3899e5f1df84aa6914c3fd46775958f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41313 } } peers { permanent_uuid: "6b8739a210f9486290e6d232c904acde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46195 } } peers { permanent_uuid: "08b0de52e3f74b048422f4308c73c9ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33631 } }
I20260619 16:43:55.472958 13642 ts_tablet_manager.cc:1428] T 31a3e455b68541ddbf94686ec4d469fb P b3899e5f1df84aa6914c3fd46775958f: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20260619 16:43:55.473415 13339 heartbeater.cc:503] Master 127.0.98.62:34179 was elected leader, sending a full tablet report...
I20260619 16:43:55.477160 13644 tablet_bootstrap.cc:492] T 31a3e455b68541ddbf94686ec4d469fb P 6b8739a210f9486290e6d232c904acde: No bootstrap required, opened a new log
I20260619 16:43:55.477409 13644 ts_tablet_manager.cc:1397] T 31a3e455b68541ddbf94686ec4d469fb P 6b8739a210f9486290e6d232c904acde: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20260619 16:43:55.482092 13644 raft_consensus.cc:348] T 31a3e455b68541ddbf94686ec4d469fb P 6b8739a210f9486290e6d232c904acde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3899e5f1df84aa6914c3fd46775958f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41313 } } peers { permanent_uuid: "6b8739a210f9486290e6d232c904acde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46195 } } peers { permanent_uuid: "08b0de52e3f74b048422f4308c73c9ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33631 } }
I20260619 16:43:55.482430 13644 raft_consensus.cc:374] T 31a3e455b68541ddbf94686ec4d469fb P 6b8739a210f9486290e6d232c904acde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:55.482477 13644 raft_consensus.cc:729] T 31a3e455b68541ddbf94686ec4d469fb P 6b8739a210f9486290e6d232c904acde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b8739a210f9486290e6d232c904acde, State: Initialized, Role: FOLLOWER
I20260619 16:43:55.482987 13644 consensus_queue.cc:260] T 31a3e455b68541ddbf94686ec4d469fb P 6b8739a210f9486290e6d232c904acde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3899e5f1df84aa6914c3fd46775958f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41313 } } peers { permanent_uuid: "6b8739a210f9486290e6d232c904acde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46195 } } peers { permanent_uuid: "08b0de52e3f74b048422f4308c73c9ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33631 } }
I20260619 16:43:55.484251 13644 ts_tablet_manager.cc:1428] T 31a3e455b68541ddbf94686ec4d469fb P 6b8739a210f9486290e6d232c904acde: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260619 16:43:55.484687 13614 heartbeater.cc:503] Master 127.0.98.62:34179 was elected leader, sending a full tablet report...
W20260619 16:43:55.490691 13340 tablet.cc:2369] T 31a3e455b68541ddbf94686ec4d469fb P b3899e5f1df84aa6914c3fd46775958f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:43:55.565876 13475 tablet.cc:2369] T 31a3e455b68541ddbf94686ec4d469fb P 08b0de52e3f74b048422f4308c73c9ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:43:55.586109 13615 tablet.cc:2369] T 31a3e455b68541ddbf94686ec4d469fb P 6b8739a210f9486290e6d232c904acde: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:43:55.622203 13648 raft_consensus.cc:482] T 31a3e455b68541ddbf94686ec4d469fb P b3899e5f1df84aa6914c3fd46775958f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:43:55.622467 13648 raft_consensus.cc:504] T 31a3e455b68541ddbf94686ec4d469fb P b3899e5f1df84aa6914c3fd46775958f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3899e5f1df84aa6914c3fd46775958f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41313 } } peers { permanent_uuid: "6b8739a210f9486290e6d232c904acde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46195 } } peers { permanent_uuid: "08b0de52e3f74b048422f4308c73c9ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33631 } }
I20260619 16:43:55.624032 13648 leader_election.cc:288] T 31a3e455b68541ddbf94686ec4d469fb P b3899e5f1df84aa6914c3fd46775958f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b8739a210f9486290e6d232c904acde (127.0.98.3:46195), 08b0de52e3f74b048422f4308c73c9ab (127.0.98.2:33631)
I20260619 16:43:55.631346 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:55.625585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57609 (local address 127.0.98.3:46195)
0619 16:43:55.625893 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:55.625899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:55.626241 (+ 342us) server_negotiation.cc:407] Connection header received
0619 16:43:55.626471 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:55.626474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:55.626530 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:55.626579 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:55.627041 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.628091 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.629922 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.630517 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.630611 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:55.630707 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:55.631100 (+ 393us) server_negotiation.cc:299] Negotiation successful
0619 16:43:55.631199 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216}
I20260619 16:43:55.632194 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:55.624504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46195 (local address 127.0.98.1:57609)
0619 16:43:55.626142 (+ 1638us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:55.626158 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:55.626333 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:55.626729 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:55.626732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:55.626748 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:55.626910 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:55.626917 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.628251 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.628254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:55.629753 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:55.629766 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.630823 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.630827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:55.630939 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:55.630963 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 16:43:55.631867 (+ 904us) client_negotiation.cc:240] Negotiation successful
0619 16:43:55.632010 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1572,"mutex_wait_us":34,"thread_start_us":163,"threads_started":1}
I20260619 16:43:55.633111 13568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a3e455b68541ddbf94686ec4d469fb" candidate_uuid: "b3899e5f1df84aa6914c3fd46775958f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b8739a210f9486290e6d232c904acde" is_pre_election: true
I20260619 16:43:55.633457 13568 raft_consensus.cc:2454] T 31a3e455b68541ddbf94686ec4d469fb P 6b8739a210f9486290e6d232c904acde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3899e5f1df84aa6914c3fd46775958f in term 0.
I20260619 16:43:55.634064 13227 leader_election.cc:302] T 31a3e455b68541ddbf94686ec4d469fb P b3899e5f1df84aa6914c3fd46775958f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b8739a210f9486290e6d232c904acde, b3899e5f1df84aa6914c3fd46775958f; no voters:
I20260619 16:43:55.634371 13648 raft_consensus.cc:2781] T 31a3e455b68541ddbf94686ec4d469fb P b3899e5f1df84aa6914c3fd46775958f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:43:55.634436 13648 raft_consensus.cc:482] T 31a3e455b68541ddbf94686ec4d469fb P b3899e5f1df84aa6914c3fd46775958f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:43:55.634470 13648 raft_consensus.cc:3037] T 31a3e455b68541ddbf94686ec4d469fb P b3899e5f1df84aa6914c3fd46775958f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:55.635960 13648 raft_consensus.cc:504] T 31a3e455b68541ddbf94686ec4d469fb P b3899e5f1df84aa6914c3fd46775958f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3899e5f1df84aa6914c3fd46775958f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41313 } } peers { permanent_uuid: "6b8739a210f9486290e6d232c904acde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46195 } } peers { permanent_uuid: "08b0de52e3f74b048422f4308c73c9ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33631 } }
I20260619 16:43:55.636337 13648 leader_election.cc:288] T 31a3e455b68541ddbf94686ec4d469fb P b3899e5f1df84aa6914c3fd46775958f [CANDIDATE]: Term 1 election: Requested vote from peers 6b8739a210f9486290e6d232c904acde (127.0.98.3:46195), 08b0de52e3f74b048422f4308c73c9ab (127.0.98.2:33631)
I20260619 16:43:55.636861 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:55.625735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39427 (local address 127.0.98.2:33631)
0619 16:43:55.625985 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:55.625989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:55.628182 (+ 2193us) server_negotiation.cc:407] Connection header received
0619 16:43:55.628664 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:55.628667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:55.628718 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:55.628774 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:55.630850 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.631573 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.634172 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.634865 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.634955 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:55.635022 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:55.636589 (+ 1567us) server_negotiation.cc:299] Negotiation successful
0619 16:43:55.636732 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260619 16:43:55.636914 13568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a3e455b68541ddbf94686ec4d469fb" candidate_uuid: "b3899e5f1df84aa6914c3fd46775958f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b8739a210f9486290e6d232c904acde"
I20260619 16:43:55.637044 13568 raft_consensus.cc:3037] T 31a3e455b68541ddbf94686ec4d469fb P 6b8739a210f9486290e6d232c904acde [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:55.637310 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:55.625016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33631 (local address 127.0.98.1:39427)
0619 16:43:55.628075 (+ 3059us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:55.628092 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:55.628523 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:55.628899 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:55.628901 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:55.628916 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:55.629517 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:55.629525 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.631750 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.631753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:55.634010 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:55.634022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.636319 (+ 2297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.636323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:55.636434 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:55.636458 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 16:43:55.637064 (+ 606us) client_negotiation.cc:240] Negotiation successful
0619 16:43:55.637153 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2987,"mutex_wait_us":19,"thread_start_us":85,"threads_started":1}
I20260619 16:43:55.638262 13568 raft_consensus.cc:2454] T 31a3e455b68541ddbf94686ec4d469fb P 6b8739a210f9486290e6d232c904acde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3899e5f1df84aa6914c3fd46775958f in term 1.
I20260619 16:43:55.638707 13425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a3e455b68541ddbf94686ec4d469fb" candidate_uuid: "b3899e5f1df84aa6914c3fd46775958f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08b0de52e3f74b048422f4308c73c9ab" is_pre_election: true
I20260619 16:43:55.638854 13420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a3e455b68541ddbf94686ec4d469fb" candidate_uuid: "b3899e5f1df84aa6914c3fd46775958f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08b0de52e3f74b048422f4308c73c9ab"
I20260619 16:43:55.639030 13425 raft_consensus.cc:2454] T 31a3e455b68541ddbf94686ec4d469fb P 08b0de52e3f74b048422f4308c73c9ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3899e5f1df84aa6914c3fd46775958f in term 0.
I20260619 16:43:55.639917 13227 leader_election.cc:302] T 31a3e455b68541ddbf94686ec4d469fb P b3899e5f1df84aa6914c3fd46775958f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b8739a210f9486290e6d232c904acde, b3899e5f1df84aa6914c3fd46775958f; no voters: 08b0de52e3f74b048422f4308c73c9ab
I20260619 16:43:55.640228 13648 raft_consensus.cc:2781] T 31a3e455b68541ddbf94686ec4d469fb P b3899e5f1df84aa6914c3fd46775958f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:55.640436 13648 raft_consensus.cc:686] T 31a3e455b68541ddbf94686ec4d469fb P b3899e5f1df84aa6914c3fd46775958f [term 1 LEADER]: Becoming Leader. State: Replica: b3899e5f1df84aa6914c3fd46775958f, State: Running, Role: LEADER
I20260619 16:43:55.640784 13648 consensus_queue.cc:237] T 31a3e455b68541ddbf94686ec4d469fb P b3899e5f1df84aa6914c3fd46775958f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3899e5f1df84aa6914c3fd46775958f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41313 } } peers { permanent_uuid: "6b8739a210f9486290e6d232c904acde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46195 } } peers { permanent_uuid: "08b0de52e3f74b048422f4308c73c9ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33631 } }
I20260619 16:43:55.644425 13141 catalog_manager.cc:5495] T 31a3e455b68541ddbf94686ec4d469fb P b3899e5f1df84aa6914c3fd46775958f reported cstate change: term changed from 0 to 1, leader changed from <none> to b3899e5f1df84aa6914c3fd46775958f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b3899e5f1df84aa6914c3fd46775958f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3899e5f1df84aa6914c3fd46775958f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b8739a210f9486290e6d232c904acde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08b0de52e3f74b048422f4308c73c9ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33631 } health_report { overall_health: UNKNOWN } } }
Jun 19 16:43:55 dist-test-slave-03cd krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:43:55.710341 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:55.704845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51914 (local address 127.0.98.1:41313)
0619 16:43:55.704978 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:55.704983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:55.705000 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:43:55.705112 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:55.705116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:55.705204 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:55.705330 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:43:55.705792 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.706572 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.707890 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.708253 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.708379 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:55.708883 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:43:55.709541 (+ 658us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:43:55.709669 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:55.710069 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 16:43:55.710161 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:43:55.720523 13420 raft_consensus.cc:3037] T 31a3e455b68541ddbf94686ec4d469fb P 08b0de52e3f74b048422f4308c73c9ab [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:55.720746 13568 raft_consensus.cc:1261] T 31a3e455b68541ddbf94686ec4d469fb P 6b8739a210f9486290e6d232c904acde [term 1 FOLLOWER]: Refusing update from remote peer b3899e5f1df84aa6914c3fd46775958f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:43:55.721603 13648 consensus_queue.cc:1035] T 31a3e455b68541ddbf94686ec4d469fb P b3899e5f1df84aa6914c3fd46775958f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b8739a210f9486290e6d232c904acde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:43:55.722091 13420 raft_consensus.cc:1261] T 31a3e455b68541ddbf94686ec4d469fb P 08b0de52e3f74b048422f4308c73c9ab [term 1 FOLLOWER]: Refusing update from remote peer b3899e5f1df84aa6914c3fd46775958f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:43:55.724974 13648 consensus_queue.cc:1035] T 31a3e455b68541ddbf94686ec4d469fb P b3899e5f1df84aa6914c3fd46775958f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08b0de52e3f74b048422f4308c73c9ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:43:55.743275 13657 mvcc.cc:205] Tried to move back new op lower bound from 7298610936701198336 to 7298610936387584000. Current Snapshot: MvccSnapshot[applied={T|T < 7298610936701198336}]
I20260619 16:43:55.754657 13658 mvcc.cc:205] Tried to move back new op lower bound from 7298610936701198336 to 7298610936387584000. Current Snapshot: MvccSnapshot[applied={T|T < 7298610936701198336}]
I20260619 16:43:55.757756 13660 mvcc.cc:205] Tried to move back new op lower bound from 7298610936701198336 to 7298610936387584000. Current Snapshot: MvccSnapshot[applied={T|T < 7298610936701198336}]
I20260619 16:43:55.774718 13139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:43:55.774932 13139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:43:55.777909 13139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0bcca7ecb21145f79c740b0641ef7a75: Sending DeleteTablet for 3 replicas of tablet 31a3e455b68541ddbf94686ec4d469fb
I20260619 16:43:55.779141 13254 tablet_service.cc:1514] Processing DeleteTablet for tablet 31a3e455b68541ddbf94686ec4d469fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51900
I20260619 16:43:55.779134 13408 tablet_service.cc:1514] Processing DeleteTablet for tablet 31a3e455b68541ddbf94686ec4d469fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43584
I20260619 16:43:55.779323 13530 tablet_service.cc:1514] Processing DeleteTablet for tablet 31a3e455b68541ddbf94686ec4d469fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34702
I20260619 16:43:55.780395 13668 tablet_replica.cc:330] T 31a3e455b68541ddbf94686ec4d469fb P 6b8739a210f9486290e6d232c904acde: stopping tablet replica
I20260619 16:43:55.780869 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 13179
I20260619 16:43:55.781531 13668 raft_consensus.cc:2229] T 31a3e455b68541ddbf94686ec4d469fb P 6b8739a210f9486290e6d232c904acde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:43:55.781930 13668 raft_consensus.cc:2258] T 31a3e455b68541ddbf94686ec4d469fb P 6b8739a210f9486290e6d232c904acde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:43:55.787926 13667 tablet_replica.cc:330] T 31a3e455b68541ddbf94686ec4d469fb P 08b0de52e3f74b048422f4308c73c9ab: stopping tablet replica
I20260619 16:43:55.788455 13667 raft_consensus.cc:2229] T 31a3e455b68541ddbf94686ec4d469fb P 08b0de52e3f74b048422f4308c73c9ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:43:55.788944 13667 raft_consensus.cc:2258] T 31a3e455b68541ddbf94686ec4d469fb P 08b0de52e3f74b048422f4308c73c9ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:43:55.789472 13668 ts_tablet_manager.cc:1905] T 31a3e455b68541ddbf94686ec4d469fb P 6b8739a210f9486290e6d232c904acde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:43:55.790616 13667 ts_tablet_manager.cc:1905] T 31a3e455b68541ddbf94686ec4d469fb P 08b0de52e3f74b048422f4308c73c9ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:43:55.792500 13668 ts_tablet_manager.cc:1918] T 31a3e455b68541ddbf94686ec4d469fb P 6b8739a210f9486290e6d232c904acde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:43:55.793009 13667 ts_tablet_manager.cc:1918] T 31a3e455b68541ddbf94686ec4d469fb P 08b0de52e3f74b048422f4308c73c9ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:43:55.793160 13667 log.cc:1196] T 31a3e455b68541ddbf94686ec4d469fb P 08b0de52e3f74b048422f4308c73c9ab: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-1/wal/wals/31a3e455b68541ddbf94686ec4d469fb
I20260619 16:43:55.793550 13667 ts_tablet_manager.cc:1939] T 31a3e455b68541ddbf94686ec4d469fb P 08b0de52e3f74b048422f4308c73c9ab: Deleting consensus metadata
I20260619 16:43:55.795130 13127 catalog_manager.cc:4841] TS 08b0de52e3f74b048422f4308c73c9ab (127.0.98.2:33631): tablet 31a3e455b68541ddbf94686ec4d469fb (table test-table [id=096374bfc4fa4d5c85a5e60ad099abe2]) successfully deleted
W20260619 16:43:55.799340 13127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41313 (error 108)
I20260619 16:43:55.801262 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:55.800905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41313 (local address 127.0.0.1:51922)
0619 16:43:55.801108 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:55.801170 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260619 16:43:55.801606 13127 catalog_manager.cc:4584] TS b3899e5f1df84aa6914c3fd46775958f (127.0.98.1:41313): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 31a3e455b68541ddbf94686ec4d469fb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41313: connect: Connection refused (error 111)
I20260619 16:43:55.801994 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 13341
I20260619 16:43:55.802582 13668 log.cc:1196] T 31a3e455b68541ddbf94686ec4d469fb P 6b8739a210f9486290e6d232c904acde: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887259574980-392-0/minicluster-data/ts-2/wal/wals/31a3e455b68541ddbf94686ec4d469fb
I20260619 16:43:55.803030 13668 ts_tablet_manager.cc:1939] T 31a3e455b68541ddbf94686ec4d469fb P 6b8739a210f9486290e6d232c904acde: Deleting consensus metadata
I20260619 16:43:55.807993 13126 catalog_manager.cc:4841] TS 6b8739a210f9486290e6d232c904acde (127.0.98.3:46195): tablet 31a3e455b68541ddbf94686ec4d469fb (table test-table [id=096374bfc4fa4d5c85a5e60ad099abe2]) successfully deleted
I20260619 16:43:55.811017 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 13477
I20260619 16:43:55.820706 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 13108
I20260619 16:43:55.829864 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 16:43:56.175885 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 16:43:56.176012 392 mini_ranger.cc:66] Stopping Ranger...
I20260619 16:43:56.522294 392 mini_ranger.cc:68] Stopped Ranger
2026-06-19 16:43:56.522 UTC [12175] LOG: received smart shutdown request
2026-06-19 16:43:56.526 UTC [12175] LOG: background worker "logical replication launcher" (PID 12184) exited with exit code 1
2026-06-19 16:43:56.527 UTC [12179] LOG: shutting down
2026-06-19 16:43:56.726 UTC [12175] LOG: database system is shut down
2026-06-19T16:43:56Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53816 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:43:56 dist-test-slave-03cd krb5kdc[13678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:43:56 dist-test-slave-03cd krb5kdc[13678](info): set up 2 sockets
Jun 19 16:43:56 dist-test-slave-03cd krb5kdc[13678](info): commencing operation
krb5kdc: starting...
W20260619 16:43:57.932001 392 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:43:58 dist-test-slave-03cd krb5kdc[13678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:43:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:43:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:43:58.067001 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38127
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38127,127.0.98.61:33449,127.0.98.60:33813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:58.188336 13691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:58.188699 13691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:58.188802 13691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:58.193212 13691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:43:58.193343 13691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:58.193378 13691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:58.193408 13691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:43:58.193446 13691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:43:58.199414 13691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38127,127.0.98.61:33449,127.0.98.60:33813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:43:58.200742 13691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:58.201897 13691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:43:58.211999 13691 server_base.cc:1034] running on GCE node
W20260619 16:43:58.211948 13697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:58.211971 13696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:58.211948 13699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:58.212915 13691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:58.214236 13691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:58.215468 13691 hybrid_clock.cc:648] HybridClock initialized: now 1781887438215450 us; error 35 us; skew 500 ppm
Jun 19 16:43:58 dist-test-slave-03cd krb5kdc[13678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:58.218654 13691 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:43:58.219871 13691 webserver.cc:460] Webserver started at http://127.0.98.62:35341/ using document root <none> and password file <none>
I20260619 16:43:58.220486 13691 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:58.220584 13691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:58.220845 13691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:58.223096 13691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "aabc55ea541b409cb877781d007d735f"
format_stamp: "Formatted at 2026-06-19 16:43:58 on dist-test-slave-03cd"
server_key: "9ccdddb1b914c479b90151e6eb349a6e"
server_key_iv: "7ccfdef8419729229eec67495b71164e"
server_key_version: "encryptionkey@0"
I20260619 16:43:58.223815 13691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "aabc55ea541b409cb877781d007d735f"
format_stamp: "Formatted at 2026-06-19 16:43:58 on dist-test-slave-03cd"
server_key: "9ccdddb1b914c479b90151e6eb349a6e"
server_key_iv: "7ccfdef8419729229eec67495b71164e"
server_key_version: "encryptionkey@0"
I20260619 16:43:58.228076 13691 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:43:58.230907 13706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:58.232434 13691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:43:58.232614 13691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "aabc55ea541b409cb877781d007d735f"
format_stamp: "Formatted at 2026-06-19 16:43:58 on dist-test-slave-03cd"
server_key: "9ccdddb1b914c479b90151e6eb349a6e"
server_key_iv: "7ccfdef8419729229eec67495b71164e"
server_key_version: "encryptionkey@0"
I20260619 16:43:58.232769 13691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:58.263589 13691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:58.267148 13691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:58.267376 13691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:58.275187 13691 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38127
I20260619 16:43:58.275206 13758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38127 every 8 connection(s)
I20260619 16:43:58.276273 13691 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:43:58.279101 13759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:58.283123 13759 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38127 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:43:58.285290 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 13691
I20260619 16:43:58.285432 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:43:58.285698 392 external_mini_cluster.cc:1428] Setting key b6e7f79b933eee53932b7bccc11eb044
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 19 16:43:58 dist-test-slave-03cd krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:43:58.302235 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.286760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44566 (local address 127.0.98.62:38127)
0619 16:43:58.287153 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:58.287165 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:58.287214 (+ 49us) server_negotiation.cc:407] Connection header received
0619 16:43:58.287927 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:58.287934 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:58.288132 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:58.288321 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:58.289196 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.290065 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.291079 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.291471 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.291644 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.296797 (+ 5153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:58.296804 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:58.296817 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:58.296841 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:58.298595 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.299129 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.299137 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.299143 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.299247 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.299565 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.299570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.299572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.300073 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:58.300229 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.300831 (+ 602us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.301113 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":287,"thread_start_us":96,"threads_started":1}
I20260619 16:43:58.302232 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.285725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38127 (local address 127.0.0.1:44566)
0619 16:43:58.286332 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.286389 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:58.287547 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:58.288321 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:58.288326 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:58.288788 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:58.289032 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.289046 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.290205 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.290208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.290950 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.290960 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.291639 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.291643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.291760 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.292409 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:58.292426 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:58.296592 (+ 4166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:58.298741 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:58.298751 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:58.298763 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:58.299002 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:58.299352 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:58.299354 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:58.299356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:58.299472 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:58.300207 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:58.300213 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:58.300493 (+ 280us) client_negotiation.cc:769] Sending connection context
0619 16:43:58.300766 (+ 273us) client_negotiation.cc:240] Negotiation successful
0619 16:43:58.301167 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":117,"thread_start_us":180,"threads_started":1}
I20260619 16:43:58.304795 13759 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33449 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:43:58.305724 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.305431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33449 (local address 127.0.0.1:55592)
0619 16:43:58.305563 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.305650 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 16:43:58.306113 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33449: connect: Connection refused (error 111)
I20260619 16:43:58.307003 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.306787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33449 (local address 127.0.0.1:55604)
0619 16:43:58.306896 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.306944 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 16:43:58.307341 13759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33449: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33449: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 16:43:58.311318 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33449
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38127,127.0.98.61:33449,127.0.98.60:33813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:43:58.356640 13759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33449 } attempt: 1
I20260619 16:43:58.357692 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.357328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33449 (local address 127.0.0.1:55606)
0619 16:43:58.357517 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.357584 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260619 16:43:58.358618 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.358325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33449 (local address 127.0.0.1:55620)
0619 16:43:58.358465 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.358523 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 16:43:58.358769 13759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33449: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33449: connect: Connection refused (error 111)
I20260619 16:43:58.427066 13759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33449 } attempt: 2
I20260619 16:43:58.428063 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.427763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33449 (local address 127.0.0.1:55634)
0619 16:43:58.427913 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.427983 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 16:43:58.428961 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.428697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33449 (local address 127.0.0.1:55646)
0619 16:43:58.428840 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.428900 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 16:43:58.429133 13759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33449: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33449: connect: Connection refused (error 111)
W20260619 16:43:58.430902 13766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:58.431295 13766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:58.431402 13766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:58.435993 13766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:43:58.436093 13766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:58.436134 13766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:58.436208 13766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:43:58.436266 13766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:43:58.442198 13766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38127,127.0.98.61:33449,127.0.98.60:33813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:43:58.443421 13766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:58.444563 13766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:43:58.453707 13766 server_base.cc:1034] running on GCE node
W20260619 16:43:58.453641 13771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:58.453670 13772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:58.453629 13774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:58.454532 13766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:58.455842 13766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:58.457032 13766 hybrid_clock.cc:648] HybridClock initialized: now 1781887438457012 us; error 37 us; skew 500 ppm
Jun 19 16:43:58 dist-test-slave-03cd krb5kdc[13678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:58.460091 13766 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 16:43:58.461160 13766 webserver.cc:460] Webserver started at http://127.0.98.61:45043/ using document root <none> and password file <none>
I20260619 16:43:58.461812 13766 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:58.461884 13766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:58.462113 13766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:58.464442 13766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-1/data/instance:
uuid: "8c1b6770fcfe460ba6eae68cb8850309"
format_stamp: "Formatted at 2026-06-19 16:43:58 on dist-test-slave-03cd"
server_key: "bc0167bbe5ead330d261fb8e340be6dd"
server_key_iv: "a9fbf8a254129342d756aafe1e909a5e"
server_key_version: "encryptionkey@0"
I20260619 16:43:58.465060 13766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-1/wal/instance:
uuid: "8c1b6770fcfe460ba6eae68cb8850309"
format_stamp: "Formatted at 2026-06-19 16:43:58 on dist-test-slave-03cd"
server_key: "bc0167bbe5ead330d261fb8e340be6dd"
server_key_iv: "a9fbf8a254129342d756aafe1e909a5e"
server_key_version: "encryptionkey@0"
I20260619 16:43:58.469183 13766 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:43:58.471961 13781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:58.473374 13766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:43:58.473546 13766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-1/wal
uuid: "8c1b6770fcfe460ba6eae68cb8850309"
format_stamp: "Formatted at 2026-06-19 16:43:58 on dist-test-slave-03cd"
server_key: "bc0167bbe5ead330d261fb8e340be6dd"
server_key_iv: "a9fbf8a254129342d756aafe1e909a5e"
server_key_version: "encryptionkey@0"
I20260619 16:43:58.473701 13766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:58.488703 13766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:58.492156 13766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:58.492388 13766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:58.499877 13766 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33449
I20260619 16:43:58.499903 13833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33449 every 8 connection(s)
I20260619 16:43:58.501093 13766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-1/data/info.pb
I20260619 16:43:58.504048 13834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:58.508080 13834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38127 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:43:58.508934 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 13766
I20260619 16:43:58.509034 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-1/wal/instance
I20260619 16:43:58.509330 392 external_mini_cluster.cc:1428] Setting key 962b4d91cfc0f91af84bd1a41e21ccf7
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260619 16:43:58.520364 13759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33449 } attempt: 3
Jun 19 16:43:58 dist-test-slave-03cd krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 16:43:58 dist-test-slave-03cd krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 16:43:58.527849 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.510357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44576 (local address 127.0.98.62:38127)
0619 16:43:58.510515 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:58.510519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:58.511784 (+ 1265us) server_negotiation.cc:407] Connection header received
0619 16:43:58.513357 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:58.513360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:58.513422 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:58.513508 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:58.515454 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.516104 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.518037 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.518302 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.518395 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.522370 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:58.522373 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:58.522376 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:58.522408 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:58.524926 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.525683 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.525688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.525690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.525796 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.526287 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.526293 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.526298 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.526555 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:58.526638 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.527448 (+ 810us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.527602 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 16:43:58.529840 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.510831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38127 (local address 127.0.0.1:44576)
0619 16:43:58.511520 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.511591 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:58.513051 (+ 1460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:58.513762 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:58.513775 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:58.514359 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:58.515212 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.515238 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.516306 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.516310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.517865 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.517883 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.518499 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.518506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.518757 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.519414 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:58.519433 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:58.522196 (+ 2763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:58.525079 (+ 2883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:58.525101 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:58.525122 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:58.525542 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:58.525921 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:58.525926 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:58.525930 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:58.526167 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:58.526727 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:58.526737 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:58.527133 (+ 396us) client_negotiation.cc:769] Sending connection context
0619 16:43:58.528053 (+ 920us) client_negotiation.cc:240] Negotiation successful
0619 16:43:58.528521 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":178,"threads_started":1}
I20260619 16:43:58.531842 13834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33449 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:43:58.532136 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.520846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33449 (local address 127.0.0.1:55652)
0619 16:43:58.520986 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.521004 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:58.521098 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:58.521850 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:58.521854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:58.522062 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:58.522225 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.522237 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.523890 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.523894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.524513 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.524526 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.525448 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.525454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.525572 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.526116 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:58.526176 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:58.528200 (+ 2024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:58.530320 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:58.530323 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:58.530325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:58.530588 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:58.530952 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:58.530954 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:58.530956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:58.531073 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:58.531601 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:58.531607 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:58.531773 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 16:43:58.531842 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 16:43:58.531975 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 16:43:58.532958 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.520991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55652 (local address 127.0.98.61:33449)
0619 16:43:58.521361 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:58.521372 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:58.521395 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:43:58.521436 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:58.521442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:58.521626 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:58.521844 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:58.522377 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.523587 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.524680 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.525285 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.525811 (+ 526us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.528357 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:58.528367 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:58.528380 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:58.528402 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:58.530181 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.530741 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.530749 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.530751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.530844 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.531190 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.531193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.531195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.531479 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:58.532478 (+ 999us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.532585 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.532764 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":154,"threads_started":1}
I20260619 16:43:58.534837 13759 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33813 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:43:58.535607 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.535356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33813 (local address 127.0.0.1:54420)
0619 16:43:58.535476 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.535531 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 16:43:58.536520 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.536199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33813 (local address 127.0.0.1:54432)
0619 16:43:58.536357 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.536451 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260619 16:43:58.536758 13759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33813: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33813: connect: Connection refused (error 111)
Jun 19 16:43:58 dist-test-slave-03cd krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 16:43:58.542273 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.532623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55658 (local address 127.0.98.61:33449)
0619 16:43:58.532933 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:58.532940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:58.532951 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:43:58.532979 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:58.532981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:58.533015 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:58.533121 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:58.533570 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.534123 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.535396 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.535635 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.535868 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.538356 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:58.538359 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:58.538361 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:58.538382 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:58.540531 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.540990 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.540993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.540995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.541119 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.541431 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.541433 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.541434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.541702 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:58.541781 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.541990 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.542118 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":253,"thread_start_us":116,"threads_started":1}
I20260619 16:43:58.542456 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.532400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33449 (local address 127.0.0.1:55658)
0619 16:43:58.532520 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.532534 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:58.532838 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:58.533216 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:58.533219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:58.533377 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:58.533460 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.533465 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.534263 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.534266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.535286 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.535296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.535866 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.535869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.535995 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.536514 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:58.536530 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:58.538177 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:58.540648 (+ 2471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:58.540651 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:58.540653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:58.540855 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:58.541251 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:58.541254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:58.541255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:58.541339 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:58.541787 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:58.541789 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:58.541872 (+ 83us) client_negotiation.cc:769] Sending connection context
0619 16:43:58.541957 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 16:43:58.542246 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":113}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 16:43:58.543479 13834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33813 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:43:58.543962 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33813
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38127,127.0.98.61:33449,127.0.98.60:33813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:43:58.544248 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.543969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33813 (local address 127.0.0.1:54438)
0619 16:43:58.544091 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.544187 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 16:43:58.544557 13785 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33813: connect: Connection refused (error 111)
I20260619 16:43:58.545554 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.545282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33813 (local address 127.0.0.1:54448)
0619 16:43:58.545417 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.545480 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 16:43:58.545794 13834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33813: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33813: connect: Connection refused (error 111)
I20260619 16:43:58.567939 13759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33813 } attempt: 1
I20260619 16:43:58.568971 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.568608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33813 (local address 127.0.0.1:54464)
0619 16:43:58.568791 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.568868 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 16:43:58.569818 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.569562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33813 (local address 127.0.0.1:54466)
0619 16:43:58.569687 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.569745 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 16:43:58.569962 13759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33813: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33813: connect: Connection refused (error 111)
I20260619 16:43:58.595125 13834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33813 } attempt: 1
I20260619 16:43:58.596133 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.595818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33813 (local address 127.0.0.1:54470)
0619 16:43:58.595965 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.596034 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 16:43:58.597049 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.596796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33813 (local address 127.0.0.1:54472)
0619 16:43:58.596938 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.596991 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260619 16:43:58.597350 13834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33813: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33813: connect: Connection refused (error 111)
I20260619 16:43:58.645143 13759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33813 } attempt: 2
I20260619 16:43:58.646085 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.645742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33813 (local address 127.0.0.1:54478)
0619 16:43:58.645934 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.646000 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 16:43:58.646890 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.646682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33813 (local address 127.0.0.1:54482)
0619 16:43:58.646787 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.646833 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 16:43:58.647089 13759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33813: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33813: connect: Connection refused (error 111)
W20260619 16:43:58.665325 13842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:58.665627 13842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:58.665671 13842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 16:43:58.665558 13834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33813 } attempt: 2
I20260619 16:43:58.666486 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.666173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33813 (local address 127.0.0.1:54498)
0619 16:43:58.666333 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.666404 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 16:43:58.667351 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.667105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33813 (local address 127.0.0.1:54500)
0619 16:43:58.667237 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.667292 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 16:43:58.667472 13834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33813: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33813: connect: Connection refused (error 111)
W20260619 16:43:58.670425 13842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:43:58.670503 13842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:58.670553 13842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:58.670603 13842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:43:58.670640 13842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:43:58.676249 13842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38127,127.0.98.61:33449,127.0.98.60:33813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:43:58.677464 13842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:58.678573 13842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:58.687387 13848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:58.687402 13850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:58.687402 13847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:58.687849 13842 server_base.cc:1034] running on GCE node
I20260619 16:43:58.688544 13842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:58.689685 13842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:58.690892 13842 hybrid_clock.cc:648] HybridClock initialized: now 1781887438690883 us; error 33 us; skew 500 ppm
Jun 19 16:43:58 dist-test-slave-03cd krb5kdc[13678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:58.693967 13842 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 16:43:58.694957 13842 webserver.cc:460] Webserver started at http://127.0.98.60:41681/ using document root <none> and password file <none>
I20260619 16:43:58.695557 13842 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:58.695652 13842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:58.695909 13842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:58.697944 13842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-2/data/instance:
uuid: "d8772eddb92942748de5ce64153c7c10"
format_stamp: "Formatted at 2026-06-19 16:43:58 on dist-test-slave-03cd"
server_key: "043e35585fc173539e0a0337cf54f0e6"
server_key_iv: "220e57c9c8a90c4da4716483ea2bb6f7"
server_key_version: "encryptionkey@0"
I20260619 16:43:58.698462 13842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-2/wal/instance:
uuid: "d8772eddb92942748de5ce64153c7c10"
format_stamp: "Formatted at 2026-06-19 16:43:58 on dist-test-slave-03cd"
server_key: "043e35585fc173539e0a0337cf54f0e6"
server_key_iv: "220e57c9c8a90c4da4716483ea2bb6f7"
server_key_version: "encryptionkey@0"
I20260619 16:43:58.702129 13842 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:43:58.704603 13857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:58.705860 13842 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:43:58.706012 13842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-2/wal
uuid: "d8772eddb92942748de5ce64153c7c10"
format_stamp: "Formatted at 2026-06-19 16:43:58 on dist-test-slave-03cd"
server_key: "043e35585fc173539e0a0337cf54f0e6"
server_key_iv: "220e57c9c8a90c4da4716483ea2bb6f7"
server_key_version: "encryptionkey@0"
I20260619 16:43:58.706132 13842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:58.730813 13842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:58.734102 13842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:58.734328 13842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:58.741946 13842 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33813
I20260619 16:43:58.741972 13909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33813 every 8 connection(s)
I20260619 16:43:58.742983 13842 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-2/data/info.pb
I20260619 16:43:58.746264 13910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:58.746285 13759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33813 } attempt: 3
I20260619 16:43:58.751209 13910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38127 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:43:58.751627 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 13842
I20260619 16:43:58.751765 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887259574980-392-0/minicluster-data/master-2/wal/instance
I20260619 16:43:58.752050 392 external_mini_cluster.cc:1428] Setting key 2e141f7275eb5979b420291de57edacc
Jun 19 16:43:58 dist-test-slave-03cd krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 16:43:58.758653 13834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33813 } attempt: 3
Jun 19 16:43:58 dist-test-slave-03cd krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 16:43:58 dist-test-slave-03cd krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:43:58.761142 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.746972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33813 (local address 127.0.0.1:54512)
0619 16:43:58.747174 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.747193 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:58.747302 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:58.750042 (+ 2740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:58.750045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:58.750276 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:58.750395 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.750403 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.751730 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.751735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.752410 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.752424 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.753383 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.753387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.753487 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.754057 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:58.754072 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:58.755559 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:58.758484 (+ 2925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:58.758488 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:58.758490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:58.758798 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:58.759214 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:58.759218 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:58.759219 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:58.759316 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:58.760179 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:58.760182 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:58.760280 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 16:43:58.760371 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 16:43:58.760955 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260619 16:43:58.763886 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.747877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54512 (local address 127.0.98.60:33813)
0619 16:43:58.748384 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:58.748399 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:58.748436 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:43:58.749340 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:58.749361 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:58.749722 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:58.750097 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:58.750570 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.751543 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.752546 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.753226 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.753838 (+ 612us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.755806 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:58.755810 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:58.755820 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:58.755844 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:58.758129 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.758930 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.758938 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.758946 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.759097 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.759433 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.759438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.759440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.760028 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:58.761609 (+ 1581us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.761792 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.762411 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":160,"threads_started":1}
I20260619 16:43:58.764178 13915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.753335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38127 (local address 127.0.0.1:44580)
0619 16:43:58.754251 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.754303 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:58.754434 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:58.754879 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:58.754886 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:58.755372 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:58.755526 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.755534 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.756516 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.756520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.757419 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.757428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.757905 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.757908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.758031 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.758487 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:58.758498 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:58.759997 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:58.762184 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:58.762193 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:58.762200 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:58.762483 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:58.762808 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:58.762811 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:58.762814 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:58.762919 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:58.763460 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:58.763467 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:58.763612 (+ 145us) client_negotiation.cc:769] Sending connection context
0619 16:43:58.763790 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 16:43:58.763978 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":83,"threads_started":1}
I20260619 16:43:58.764861 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.753419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44580 (local address 127.0.98.62:38127)
0619 16:43:58.753528 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:58.753531 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:58.754411 (+ 880us) server_negotiation.cc:407] Connection header received
0619 16:43:58.754561 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:58.754566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:58.754617 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:58.754685 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:58.755699 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.756403 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.757569 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.757790 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.757901 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.760198 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:58.760202 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:58.760204 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:58.760227 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:58.762062 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.762595 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.762600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.762603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.762690 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.763028 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.763031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.763033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.763320 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:58.763410 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.764550 (+ 1140us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.764691 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 16:43:58 dist-test-slave-03cd krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 16:43:58.766664 13910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33449 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:43:58.766811 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.752705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54526 (local address 127.0.98.60:33813)
0619 16:43:58.753187 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:58.753196 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:58.753211 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:43:58.753348 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:58.753351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:58.753403 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:58.753488 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:58.754501 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.755074 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.756144 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.756505 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.756602 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.760142 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:58.760146 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:58.760149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:58.760176 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:58.763287 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.763823 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.763829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.763834 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.763927 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.764265 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.764269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.764270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.765563 (+ 1293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:58.765753 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.766469 (+ 716us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.766599 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":164,"threads_started":1}
I20260619 16:43:58.769577 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.759164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33813 (local address 127.0.0.1:54532)
0619 16:43:58.759304 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.759321 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:58.759401 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:58.760518 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:58.760524 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:58.760719 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:58.760845 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.760859 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.761873 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.761876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.762674 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.762686 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.763287 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.763291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.763369 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.763878 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:58.763892 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:58.765333 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:58.767835 (+ 2502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:58.767839 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:58.767844 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:58.768125 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:58.768452 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:58.768457 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:58.768461 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:58.768557 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:58.769025 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:58.769028 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:58.769165 (+ 137us) client_negotiation.cc:769] Sending connection context
0619 16:43:58.769263 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 16:43:58.769415 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 16:43:58.769712 13759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f: Bootstrap starting.
I20260619 16:43:58.770442 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.759534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54532 (local address 127.0.98.60:33813)
0619 16:43:58.760296 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:58.760306 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:58.760317 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:43:58.760351 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:58.760355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:58.760401 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:58.760489 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:58.760987 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.761748 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.762831 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.763154 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.763588 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.765484 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:58.765487 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:58.765490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:58.765513 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:58.767158 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.768255 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.768258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.768260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.768347 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.768676 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.768681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.768683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.768918 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:58.770093 (+ 1175us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.770166 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.770279 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":132,"threads_started":1}
Jun 19 16:43:58 dist-test-slave-03cd krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 16:43:58.773324 13759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:58.773877 13834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c1b6770fcfe460ba6eae68cb8850309: Bootstrap starting.
Jun 19 16:43:58 dist-test-slave-03cd krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:43:58.774261 13759 log.cc:824] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:58.776871 13834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c1b6770fcfe460ba6eae68cb8850309: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:58.778025 13834 log.cc:824] T 00000000000000000000000000000000 P 8c1b6770fcfe460ba6eae68cb8850309: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:58.778242 13915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.767388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33449 (local address 127.0.0.1:55662)
0619 16:43:58.767550 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.767564 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:58.767636 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:58.767973 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:58.767976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:58.768166 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:58.768277 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.768284 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.769413 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.769416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.770143 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.770153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.770835 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.770839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.770923 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.771794 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:58.771808 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:58.773885 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:58.776032 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:58.776035 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:58.776038 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:58.776316 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:58.776682 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:58.776689 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:58.776693 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:58.776788 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:58.777607 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:58.777610 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:58.777773 (+ 163us) client_negotiation.cc:769] Sending connection context
0619 16:43:58.777907 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 16:43:58.778074 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 16:43:58.778323 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.767463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55662 (local address 127.0.98.61:33449)
0619 16:43:58.767684 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:58.767688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:58.767702 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:43:58.767744 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:58.767747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:58.767791 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:58.767866 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:58.768414 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.769193 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.770278 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.770609 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.770688 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.774068 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:58.774073 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:58.774076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:58.774103 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:58.775891 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.776432 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.776439 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.776443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.776535 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.776898 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.776902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.776903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.777212 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:58.777552 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.777976 (+ 424us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.778169 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260619 16:43:58.778749 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.768222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44592 (local address 127.0.98.62:38127)
0619 16:43:58.768340 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:58.768344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:58.768511 (+ 167us) server_negotiation.cc:407] Connection header received
0619 16:43:58.768701 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:58.768704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:58.768752 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:58.768803 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:58.769681 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.770404 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.771465 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.771789 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.771894 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.774768 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:58.774774 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:58.774780 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:58.774813 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:58.776404 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.776921 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.776925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.776927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.777035 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.777576 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.777579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.777581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.777844 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:58.778043 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.778458 (+ 415us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.778598 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 16:43:58.779156 13910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33813 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:43:58.781164 13834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c1b6770fcfe460ba6eae68cb8850309: No bootstrap required, opened a new log
I20260619 16:43:58.781172 13759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f: No bootstrap required, opened a new log
Jun 19 16:43:58 dist-test-slave-03cd krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 16:43:58.785647 13759 raft_consensus.cc:348] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabc55ea541b409cb877781d007d735f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38127 } } peers { permanent_uuid: "8c1b6770fcfe460ba6eae68cb8850309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33449 } } peers { permanent_uuid: "d8772eddb92942748de5ce64153c7c10" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33813 } }
I20260619 16:43:58.785647 13834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c1b6770fcfe460ba6eae68cb8850309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabc55ea541b409cb877781d007d735f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38127 } } peers { permanent_uuid: "8c1b6770fcfe460ba6eae68cb8850309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33449 } } peers { permanent_uuid: "d8772eddb92942748de5ce64153c7c10" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33813 } }
I20260619 16:43:58.785928 13834 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c1b6770fcfe460ba6eae68cb8850309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:58.785984 13834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c1b6770fcfe460ba6eae68cb8850309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1b6770fcfe460ba6eae68cb8850309, State: Initialized, Role: FOLLOWER
I20260619 16:43:58.786332 13759 raft_consensus.cc:374] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:58.786401 13759 raft_consensus.cc:729] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aabc55ea541b409cb877781d007d735f, State: Initialized, Role: FOLLOWER
I20260619 16:43:58.786517 13834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c1b6770fcfe460ba6eae68cb8850309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabc55ea541b409cb877781d007d735f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38127 } } peers { permanent_uuid: "8c1b6770fcfe460ba6eae68cb8850309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33449 } } peers { permanent_uuid: "d8772eddb92942748de5ce64153c7c10" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33813 } }
I20260619 16:43:58.786825 13759 consensus_queue.cc:260] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabc55ea541b409cb877781d007d735f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38127 } } peers { permanent_uuid: "8c1b6770fcfe460ba6eae68cb8850309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33449 } } peers { permanent_uuid: "d8772eddb92942748de5ce64153c7c10" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33813 } }
I20260619 16:43:58.787356 13920 sys_catalog.cc:455] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabc55ea541b409cb877781d007d735f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38127 } } peers { permanent_uuid: "8c1b6770fcfe460ba6eae68cb8850309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33449 } } peers { permanent_uuid: "d8772eddb92942748de5ce64153c7c10" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33813 } } }
I20260619 16:43:58.787592 13920 sys_catalog.cc:458] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:43:58.788142 13759 sys_catalog.cc:564] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:43:58.788143 13834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c1b6770fcfe460ba6eae68cb8850309 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:43:58.788345 13919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c1b6770fcfe460ba6eae68cb8850309 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabc55ea541b409cb877781d007d735f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38127 } } peers { permanent_uuid: "8c1b6770fcfe460ba6eae68cb8850309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33449 } } peers { permanent_uuid: "d8772eddb92942748de5ce64153c7c10" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33813 } } }
I20260619 16:43:58.788537 13919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c1b6770fcfe460ba6eae68cb8850309 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:43:58.789892 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.780523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55668 (local address 127.0.98.61:33449)
0619 16:43:58.780627 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:58.780631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:58.780799 (+ 168us) server_negotiation.cc:407] Connection header received
0619 16:43:58.780955 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:58.780958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:58.781001 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:58.781091 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:58.781764 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.782271 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.783034 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.783273 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.783385 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.785970 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:58.785974 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:58.785977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:58.786016 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:58.787779 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.788264 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.788267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.788269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.788367 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.788715 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.788718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.788720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.788916 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:58.789145 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.789663 (+ 518us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.789778 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 16:43:58.793757 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 16:43:58.793864 13941 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:43:58.793926 13941 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 19 16:43:58 dist-test-slave-03cd krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 16:43:58.800288 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.780004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54548 (local address 127.0.98.60:33813)
0619 16:43:58.780119 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:58.780123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:58.790977 (+ 10854us) server_negotiation.cc:407] Connection header received
0619 16:43:58.791145 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:58.791150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:58.791195 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:58.791244 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:58.791832 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.792587 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.793581 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.793931 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.794010 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.796716 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:58.796719 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:58.796722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:58.796746 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:58.798450 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.798995 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.799000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.799003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.799082 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.799337 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.799339 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.799341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.799550 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:58.799630 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.799994 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.800126 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 16:43:58.800985 13915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.779778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33813 (local address 127.0.0.1:54548)
0619 16:43:58.790898 (+ 11120us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.790916 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:58.791046 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:58.791385 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:58.791387 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:58.791583 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:58.791711 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.791720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.792732 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.792735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.793459 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.793470 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.794178 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.794182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.794257 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.794975 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:58.794991 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:58.796562 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:58.798587 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:58.798591 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:58.798594 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:58.798884 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:58.799176 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:58.799178 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:58.799180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:58.799254 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:58.799715 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:58.799718 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:58.799831 (+ 113us) client_negotiation.cc:769] Sending connection context
0619 16:43:58.800685 (+ 854us) client_negotiation.cc:240] Negotiation successful
0619 16:43:58.800831 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10995}
I20260619 16:43:58.796564 13691 master_runner.cc:186] Error getting master registration for 127.0.98.60:33813: OK, instance_id { permanent_uuid: "d8772eddb92942748de5ce64153c7c10" instance_seqno: 1781887438739811 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:43:58.801179 13691 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 16:43:58.802626 13942 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8c1b6770fcfe460ba6eae68cb8850309: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:43:58.802802 13942 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:43:58.804576 13910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8772eddb92942748de5ce64153c7c10: Bootstrap starting.
I20260619 16:43:58.801576 13766 master_runner.cc:186] Error getting master registration for 127.0.98.60:33813: OK, instance_id { permanent_uuid: "d8772eddb92942748de5ce64153c7c10" instance_seqno: 1781887438739811 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:43:58.804908 13766 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:43:58.807397 13910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8772eddb92942748de5ce64153c7c10: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:58.808408 13910 log.cc:824] T 00000000000000000000000000000000 P d8772eddb92942748de5ce64153c7c10: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:58.811257 13910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8772eddb92942748de5ce64153c7c10: No bootstrap required, opened a new log
I20260619 16:43:58.818601 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.798031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55684 (local address 127.0.98.61:33449)
0619 16:43:58.798159 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:58.798163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:58.800247 (+ 2084us) server_negotiation.cc:407] Connection header received
0619 16:43:58.800285 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:58.800289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:58.800340 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:58.800411 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:58.809891 (+ 9480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.810720 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.812050 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.812396 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.812484 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.814528 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:58.814536 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:58.814542 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:58.814607 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:58.816660 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.817121 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.817127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.817129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.817236 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.817547 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.817550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.817552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.817790 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:58.817903 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.818285 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.818424 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 16:43:58.819451 13910 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8772eddb92942748de5ce64153c7c10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabc55ea541b409cb877781d007d735f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38127 } } peers { permanent_uuid: "8c1b6770fcfe460ba6eae68cb8850309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33449 } } peers { permanent_uuid: "d8772eddb92942748de5ce64153c7c10" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33813 } }
I20260619 16:43:58.819870 13910 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8772eddb92942748de5ce64153c7c10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:58.819952 13910 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8772eddb92942748de5ce64153c7c10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8772eddb92942748de5ce64153c7c10, State: Initialized, Role: FOLLOWER
I20260619 16:43:58.820658 13910 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8772eddb92942748de5ce64153c7c10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabc55ea541b409cb877781d007d735f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38127 } } peers { permanent_uuid: "8c1b6770fcfe460ba6eae68cb8850309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33449 } } peers { permanent_uuid: "d8772eddb92942748de5ce64153c7c10" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33813 } }
I20260619 16:43:58.821600 13953 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8772eddb92942748de5ce64153c7c10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabc55ea541b409cb877781d007d735f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38127 } } peers { permanent_uuid: "8c1b6770fcfe460ba6eae68cb8850309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33449 } } peers { permanent_uuid: "d8772eddb92942748de5ce64153c7c10" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33813 } } }
I20260619 16:43:58.821851 13953 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8772eddb92942748de5ce64153c7c10 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:43:58.822836 13910 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8772eddb92942748de5ce64153c7c10 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:43:58.825898 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.798649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54560 (local address 127.0.98.60:33813)
0619 16:43:58.798759 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:58.798764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:58.800209 (+ 1445us) server_negotiation.cc:407] Connection header received
0619 16:43:58.800375 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:58.800381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:58.800429 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:58.800475 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:58.802885 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.803692 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.806653 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.807007 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.807087 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.812312 (+ 5225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:58.812317 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:58.812320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:58.812351 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:58.814870 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.823819 (+ 8949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.823824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.823826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.823985 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.824761 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.824766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.824768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.824993 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:58.825116 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.825454 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.825600 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:43:58.826160 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.799280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44604 (local address 127.0.98.62:38127)
0619 16:43:58.799381 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:58.799386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:58.801666 (+ 2280us) server_negotiation.cc:407] Connection header received
0619 16:43:58.801843 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:58.801846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:58.801897 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:58.801950 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:58.804814 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.805507 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.807722 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.808046 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.808126 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.812751 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:58.812756 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:58.812758 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:58.812791 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:58.815806 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.823820 (+ 8014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.823825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.823826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.824141 (+ 315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.824547 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.824552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.824554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.824846 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:58.824997 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.825837 (+ 840us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.825990 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 16:43:58.832660 13842 master_runner.cc:263] No leader master found from master d8772eddb92942748de5ce64153c7c10
I20260619 16:43:58.832763 13842 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 16:43:58.835774 13964 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d8772eddb92942748de5ce64153c7c10: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:43:58.835891 13964 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:43:58.952405 13920 raft_consensus.cc:482] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:43:58.952641 13920 raft_consensus.cc:504] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabc55ea541b409cb877781d007d735f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38127 } } peers { permanent_uuid: "8c1b6770fcfe460ba6eae68cb8850309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33449 } } peers { permanent_uuid: "d8772eddb92942748de5ce64153c7c10" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33813 } }
I20260619 16:43:58.953797 13920 leader_election.cc:288] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c1b6770fcfe460ba6eae68cb8850309 (127.0.98.61:33449), d8772eddb92942748de5ce64153c7c10 (127.0.98.60:33813)
I20260619 16:43:58.954888 13884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aabc55ea541b409cb877781d007d735f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8772eddb92942748de5ce64153c7c10" is_pre_election: true
I20260619 16:43:58.955272 13884 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d8772eddb92942748de5ce64153c7c10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aabc55ea541b409cb877781d007d735f in term 0.
I20260619 16:43:58.956193 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aabc55ea541b409cb877781d007d735f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c1b6770fcfe460ba6eae68cb8850309" is_pre_election: true
I20260619 16:43:58.956517 13808 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8c1b6770fcfe460ba6eae68cb8850309 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aabc55ea541b409cb877781d007d735f in term 0.
I20260619 16:43:58.957154 13710 leader_election.cc:302] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c1b6770fcfe460ba6eae68cb8850309, aabc55ea541b409cb877781d007d735f; no voters:
I20260619 16:43:58.957861 13920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:43:58.957959 13920 raft_consensus.cc:482] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:43:58.958006 13920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:58.959493 13920 raft_consensus.cc:504] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabc55ea541b409cb877781d007d735f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38127 } } peers { permanent_uuid: "8c1b6770fcfe460ba6eae68cb8850309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33449 } } peers { permanent_uuid: "d8772eddb92942748de5ce64153c7c10" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33813 } }
I20260619 16:43:58.960212 13920 leader_election.cc:288] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f [CANDIDATE]: Term 1 election: Requested vote from peers 8c1b6770fcfe460ba6eae68cb8850309 (127.0.98.61:33449), d8772eddb92942748de5ce64153c7c10 (127.0.98.60:33813)
I20260619 16:43:58.960863 13884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aabc55ea541b409cb877781d007d735f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8772eddb92942748de5ce64153c7c10"
I20260619 16:43:58.961016 13884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8772eddb92942748de5ce64153c7c10 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:58.962460 13884 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d8772eddb92942748de5ce64153c7c10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aabc55ea541b409cb877781d007d735f in term 1.
I20260619 16:43:58.963171 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aabc55ea541b409cb877781d007d735f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c1b6770fcfe460ba6eae68cb8850309"
I20260619 16:43:58.963322 13808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c1b6770fcfe460ba6eae68cb8850309 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:58.964805 13808 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8c1b6770fcfe460ba6eae68cb8850309 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aabc55ea541b409cb877781d007d735f in term 1.
I20260619 16:43:58.965339 13710 leader_election.cc:302] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c1b6770fcfe460ba6eae68cb8850309, aabc55ea541b409cb877781d007d735f; no voters:
I20260619 16:43:58.965785 13920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:58.966099 13920 raft_consensus.cc:686] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f [term 1 LEADER]: Becoming Leader. State: Replica: aabc55ea541b409cb877781d007d735f, State: Running, Role: LEADER
I20260619 16:43:58.966539 13920 consensus_queue.cc:237] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabc55ea541b409cb877781d007d735f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38127 } } peers { permanent_uuid: "8c1b6770fcfe460ba6eae68cb8850309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33449 } } peers { permanent_uuid: "d8772eddb92942748de5ce64153c7c10" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33813 } }
I20260619 16:43:58.969127 13920 sys_catalog.cc:455] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f [sys.catalog]: SysCatalogTable state changed. Reason: New leader aabc55ea541b409cb877781d007d735f. Latest consensus state: current_term: 1 leader_uuid: "aabc55ea541b409cb877781d007d735f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabc55ea541b409cb877781d007d735f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38127 } } peers { permanent_uuid: "8c1b6770fcfe460ba6eae68cb8850309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33449 } } peers { permanent_uuid: "d8772eddb92942748de5ce64153c7c10" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33813 } } }
I20260619 16:43:58.969614 13920 sys_catalog.cc:458] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:58.970155 13968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:43:58.971563 13968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:43:58.977924 13884 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d8772eddb92942748de5ce64153c7c10 [term 1 FOLLOWER]: Refusing update from remote peer aabc55ea541b409cb877781d007d735f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:43:58.978909 13966 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8772eddb92942748de5ce64153c7c10" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:43:58.985946 13863 tablet.cc:2369] T 00000000000000000000000000000000 P d8772eddb92942748de5ce64153c7c10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:43:58.987090 13953 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8772eddb92942748de5ce64153c7c10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aabc55ea541b409cb877781d007d735f. Latest consensus state: current_term: 1 leader_uuid: "aabc55ea541b409cb877781d007d735f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabc55ea541b409cb877781d007d735f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38127 } } peers { permanent_uuid: "8c1b6770fcfe460ba6eae68cb8850309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33449 } } peers { permanent_uuid: "d8772eddb92942748de5ce64153c7c10" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33813 } } }
I20260619 16:43:58.987242 13953 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8772eddb92942748de5ce64153c7c10 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:43:58.989738 13808 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8c1b6770fcfe460ba6eae68cb8850309 [term 1 FOLLOWER]: Refusing update from remote peer aabc55ea541b409cb877781d007d735f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:43:58.990671 13966 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c1b6770fcfe460ba6eae68cb8850309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:43:58.994010 13787 tablet.cc:2369] T 00000000000000000000000000000000 P 8c1b6770fcfe460ba6eae68cb8850309: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:43:58.995651 13919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c1b6770fcfe460ba6eae68cb8850309 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aabc55ea541b409cb877781d007d735f. Latest consensus state: current_term: 1 leader_uuid: "aabc55ea541b409cb877781d007d735f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabc55ea541b409cb877781d007d735f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38127 } } peers { permanent_uuid: "8c1b6770fcfe460ba6eae68cb8850309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33449 } } peers { permanent_uuid: "d8772eddb92942748de5ce64153c7c10" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33813 } } }
I20260619 16:43:58.995863 13919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c1b6770fcfe460ba6eae68cb8850309 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:43:59.004810 13966 sys_catalog.cc:455] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aabc55ea541b409cb877781d007d735f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabc55ea541b409cb877781d007d735f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38127 } } peers { permanent_uuid: "8c1b6770fcfe460ba6eae68cb8850309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33449 } } peers { permanent_uuid: "d8772eddb92942748de5ce64153c7c10" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33813 } } }
I20260619 16:43:59.004974 13966 sys_catalog.cc:458] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:59.007741 13969 mvcc.cc:205] Tried to move back new op lower bound from 7298610950045831168 to 7298610950012874752. Current Snapshot: MvccSnapshot[applied={T|T < 7298610950045831168}]
I20260619 16:43:59.009328 13919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c1b6770fcfe460ba6eae68cb8850309 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aabc55ea541b409cb877781d007d735f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabc55ea541b409cb877781d007d735f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38127 } } peers { permanent_uuid: "8c1b6770fcfe460ba6eae68cb8850309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33449 } } peers { permanent_uuid: "d8772eddb92942748de5ce64153c7c10" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33813 } } }
I20260619 16:43:59.009486 13919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c1b6770fcfe460ba6eae68cb8850309 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:43:59.010059 13968 catalog_manager.cc:1347] Generated new cluster ID: 8789a6fb9d5e48918ef16c319061ed6d
I20260619 16:43:59.010129 13968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:43:59.010637 13972 mvcc.cc:205] Tried to move back new op lower bound from 7298610950045831168 to 7298610950012874752. Current Snapshot: MvccSnapshot[applied={T|T < 7298610950045831168}]
I20260619 16:43:59.013458 13920 sys_catalog.cc:455] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aabc55ea541b409cb877781d007d735f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabc55ea541b409cb877781d007d735f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38127 } } peers { permanent_uuid: "8c1b6770fcfe460ba6eae68cb8850309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33449 } } peers { permanent_uuid: "d8772eddb92942748de5ce64153c7c10" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33813 } } }
I20260619 16:43:59.013633 13920 sys_catalog.cc:458] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:59.015507 13953 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8772eddb92942748de5ce64153c7c10 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aabc55ea541b409cb877781d007d735f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabc55ea541b409cb877781d007d735f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38127 } } peers { permanent_uuid: "8c1b6770fcfe460ba6eae68cb8850309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33449 } } peers { permanent_uuid: "d8772eddb92942748de5ce64153c7c10" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33813 } } }
I20260619 16:43:59.016294 13953 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8772eddb92942748de5ce64153c7c10 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:43:59.036541 13968 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:43:59.038597 13968 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:43:59.057132 13968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aabc55ea541b409cb877781d007d735f: Generated new TSK 0
I20260619 16:43:59.058108 13968 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:43:59.804325 13942 catalog_manager.cc:1259] Loaded cluster ID: 8789a6fb9d5e48918ef16c319061ed6d
I20260619 16:43:59.804430 13942 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8c1b6770fcfe460ba6eae68cb8850309: loading cluster ID for follower catalog manager: success
I20260619 16:43:59.806165 13942 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8c1b6770fcfe460ba6eae68cb8850309: acquiring CA information for follower catalog manager: success
I20260619 16:43:59.807394 13942 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8c1b6770fcfe460ba6eae68cb8850309: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:43:59.837466 13964 catalog_manager.cc:1259] Loaded cluster ID: 8789a6fb9d5e48918ef16c319061ed6d
I20260619 16:43:59.837604 13964 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d8772eddb92942748de5ce64153c7c10: loading cluster ID for follower catalog manager: success
I20260619 16:43:59.839524 13964 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d8772eddb92942748de5ce64153c7c10: acquiring CA information for follower catalog manager: success
I20260619 16:43:59.840891 13964 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d8772eddb92942748de5ce64153c7c10: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:44:00.129396 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 13691
I20260619 16:44:00.135632 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 13766
I20260619 16:44:00.141924 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 13842
2026-06-19T16:44:00Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3277 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:44:00 dist-test-slave-03cd krb5kdc[13984](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:44:00 dist-test-slave-03cd krb5kdc[13984](info): set up 2 sockets
Jun 19 16:44:00 dist-test-slave-03cd krb5kdc[13984](info): commencing operation
krb5kdc: starting...
W20260619 16:44:01.220052 392 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:44:01 dist-test-slave-03cd krb5kdc[13984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:44:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:44:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:44:01.355786 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42003
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:44:01.496837 13997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:01.497262 13997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:01.497346 13997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:01.501906 13997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:44:01.502038 13997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:01.502079 13997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:01.502122 13997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:44:01.502156 13997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:44:01.508479 13997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:44:01.510087 13997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:01.511442 13997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:44:01.522356 13997 server_base.cc:1034] running on GCE node
W20260619 16:44:01.522295 14005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:01.522295 14003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:01.522298 14002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:01.523600 13997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:01.525061 13997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:01.526326 13997 hybrid_clock.cc:648] HybridClock initialized: now 1781887441526281 us; error 69 us; skew 500 ppm
Jun 19 16:44:01 dist-test-slave-03cd krb5kdc[13984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:01.529888 13997 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:44:01.531026 13997 webserver.cc:460] Webserver started at http://127.0.98.62:44865/ using document root <none> and password file <none>
I20260619 16:44:01.531647 13997 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:01.531796 13997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:01.532009 13997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:44:01.534385 13997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "45fee2aa3032434795740b46de5a8d4f"
format_stamp: "Formatted at 2026-06-19 16:44:01 on dist-test-slave-03cd"
server_key: "84f8620278b6b6ef9d1be477c310cf6e"
server_key_iv: "8947bb5104dd3edd359375eb7d38387a"
server_key_version: "encryptionkey@0"
I20260619 16:44:01.535008 13997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "45fee2aa3032434795740b46de5a8d4f"
format_stamp: "Formatted at 2026-06-19 16:44:01 on dist-test-slave-03cd"
server_key: "84f8620278b6b6ef9d1be477c310cf6e"
server_key_iv: "8947bb5104dd3edd359375eb7d38387a"
server_key_version: "encryptionkey@0"
I20260619 16:44:01.540624 13997 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260619 16:44:01.544034 14012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:01.545809 13997 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:44:01.546020 13997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "45fee2aa3032434795740b46de5a8d4f"
format_stamp: "Formatted at 2026-06-19 16:44:01 on dist-test-slave-03cd"
server_key: "84f8620278b6b6ef9d1be477c310cf6e"
server_key_iv: "8947bb5104dd3edd359375eb7d38387a"
server_key_version: "encryptionkey@0"
I20260619 16:44:01.546183 13997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:01.575495 13997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:01.579228 13997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:01.579491 13997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:01.588126 13997 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42003
I20260619 16:44:01.588176 14064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42003 every 8 connection(s)
I20260619 16:44:01.589315 13997 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:44:01.593230 14065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:01.595381 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 13997
I20260619 16:44:01.595537 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:44:01.595916 392 external_mini_cluster.cc:1428] Setting key aed24828529c9cc5b731ce5de93ae544
I20260619 16:44:01.601928 14065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45fee2aa3032434795740b46de5a8d4f: Bootstrap starting.
I20260619 16:44:01.605851 14065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45fee2aa3032434795740b46de5a8d4f: Neither blocks nor log segments found. Creating new log.
Jun 19 16:44:01 dist-test-slave-03cd krb5kdc[13984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:44:01.606940 14065 log.cc:824] T 00000000000000000000000000000000 P 45fee2aa3032434795740b46de5a8d4f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:44:01.610258 14065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45fee2aa3032434795740b46de5a8d4f: No bootstrap required, opened a new log
I20260619 16:44:01.613327 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:01.597846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49810 (local address 127.0.98.62:42003)
0619 16:44:01.598483 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:01.598503 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:01.598551 (+ 48us) server_negotiation.cc:407] Connection header received
0619 16:44:01.599416 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:01.599439 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:01.599968 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:01.600420 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:01.601349 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:01.602506 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:01.603420 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:01.604059 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:01.604259 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:01.607364 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:01.607374 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:01.607401 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:01.607455 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:01.609847 (+ 2392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:01.610447 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:01.610455 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:01.610463 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:01.610647 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:01.610956 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:01.610959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:01.610961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:01.611503 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:01.611777 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:01.612098 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 16:44:01.612455 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":188,"threads_started":1}
I20260619 16:44:01.615091 14065 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45fee2aa3032434795740b46de5a8d4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fee2aa3032434795740b46de5a8d4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42003 } }
I20260619 16:44:01.615475 14065 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45fee2aa3032434795740b46de5a8d4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:01.615533 14065 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45fee2aa3032434795740b46de5a8d4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45fee2aa3032434795740b46de5a8d4f, State: Initialized, Role: FOLLOWER
I20260619 16:44:01.616250 14065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45fee2aa3032434795740b46de5a8d4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fee2aa3032434795740b46de5a8d4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42003 } }
I20260619 16:44:01.616437 14065 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45fee2aa3032434795740b46de5a8d4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:44:01.616515 14065 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45fee2aa3032434795740b46de5a8d4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:44:01.616632 14065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45fee2aa3032434795740b46de5a8d4f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:01.617431 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:44:01.618291 14065 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45fee2aa3032434795740b46de5a8d4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fee2aa3032434795740b46de5a8d4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42003 } }
I20260619 16:44:01.618773 14065 leader_election.cc:302] T 00000000000000000000000000000000 P 45fee2aa3032434795740b46de5a8d4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45fee2aa3032434795740b46de5a8d4f; no voters:
I20260619 16:44:01.619220 14065 leader_election.cc:288] T 00000000000000000000000000000000 P 45fee2aa3032434795740b46de5a8d4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:44:01.619407 14070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45fee2aa3032434795740b46de5a8d4f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:44:01.619998 14070 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45fee2aa3032434795740b46de5a8d4f [term 1 LEADER]: Becoming Leader. State: Replica: 45fee2aa3032434795740b46de5a8d4f, State: Running, Role: LEADER
I20260619 16:44:01.620425 14070 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45fee2aa3032434795740b46de5a8d4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fee2aa3032434795740b46de5a8d4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42003 } }
I20260619 16:44:01.620968 14065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45fee2aa3032434795740b46de5a8d4f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:44:01.622478 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45fee2aa3032434795740b46de5a8d4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45fee2aa3032434795740b46de5a8d4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fee2aa3032434795740b46de5a8d4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42003 } } }
I20260619 16:44:01.622673 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45fee2aa3032434795740b46de5a8d4f [sys.catalog]: This master's current role is: LEADER
I20260619 16:44:01.623036 14072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45fee2aa3032434795740b46de5a8d4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45fee2aa3032434795740b46de5a8d4f. Latest consensus state: current_term: 1 leader_uuid: "45fee2aa3032434795740b46de5a8d4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fee2aa3032434795740b46de5a8d4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42003 } } }
I20260619 16:44:01.623158 14072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45fee2aa3032434795740b46de5a8d4f [sys.catalog]: This master's current role is: LEADER
W20260619 16:44:01.626880 14085 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 45fee2aa3032434795740b46de5a8d4f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:44:01.627012 14085 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:44:01.627125 14080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:44:01.628065 14080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:44:01.634199 14080 catalog_manager.cc:1347] Generated new cluster ID: d9b285ce7778454e8e249b95c27d16a0
I20260619 16:44:01.634305 14080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:44:01.641523 14080 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:44:01.643321 14080 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:44:01.651878 14080 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45fee2aa3032434795740b46de5a8d4f: Generated new TSK 0
I20260619 16:44:01.652817 14080 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:44:01.675288 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 13997
2026-06-19T16:44:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1533 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:44:01 dist-test-slave-03cd krb5kdc[14092](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:44:01 dist-test-slave-03cd krb5kdc[14092](info): set up 2 sockets
Jun 19 16:44:01 dist-test-slave-03cd krb5kdc[14092](info): commencing operation
krb5kdc: starting...
W20260619 16:44:02.748100 392 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:44:02 dist-test-slave-03cd krb5kdc[14092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:44:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:44:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:44:02.883780 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34767
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:44:03.010830 14105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:03.011160 14105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:03.011207 14105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:03.015553 14105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:44:03.015686 14105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:03.015712 14105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:03.015733 14105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:44:03.015753 14105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:44:03.022043 14105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:44:03.023260 14105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:03.024507 14105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:44:03.035051 14105 server_base.cc:1034] running on GCE node
W20260619 16:44:03.034986 14111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:03.034986 14113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:03.035036 14110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:03.036003 14105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:03.037320 14105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:03.038514 14105 hybrid_clock.cc:648] HybridClock initialized: now 1781887443038481 us; error 53 us; skew 500 ppm
Jun 19 16:44:03 dist-test-slave-03cd krb5kdc[14092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:03.045871 14105 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:44:03.047098 14105 webserver.cc:460] Webserver started at http://127.0.98.62:37499/ using document root <none> and password file <none>
I20260619 16:44:03.047878 14105 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:03.047978 14105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:03.048225 14105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:44:03.050519 14105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "e06d30f6f6d24164b9db02ecfed70331"
format_stamp: "Formatted at 2026-06-19 16:44:03 on dist-test-slave-03cd"
server_key: "baa2404c716818f086beba88a69cfdfa"
server_key_iv: "7d884047083ab07b9f332bf4a7b5519f"
server_key_version: "encryptionkey@0"
I20260619 16:44:03.051120 14105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "e06d30f6f6d24164b9db02ecfed70331"
format_stamp: "Formatted at 2026-06-19 16:44:03 on dist-test-slave-03cd"
server_key: "baa2404c716818f086beba88a69cfdfa"
server_key_iv: "7d884047083ab07b9f332bf4a7b5519f"
server_key_version: "encryptionkey@0"
I20260619 16:44:03.055125 14105 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260619 16:44:03.058084 14120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:03.059621 14105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:44:03.059861 14105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "e06d30f6f6d24164b9db02ecfed70331"
format_stamp: "Formatted at 2026-06-19 16:44:03 on dist-test-slave-03cd"
server_key: "baa2404c716818f086beba88a69cfdfa"
server_key_iv: "7d884047083ab07b9f332bf4a7b5519f"
server_key_version: "encryptionkey@0"
I20260619 16:44:03.059981 14105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:03.079015 14105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:03.082674 14105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:03.082912 14105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:03.090574 14105 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34767
I20260619 16:44:03.090646 14172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34767 every 8 connection(s)
I20260619 16:44:03.091645 14105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:44:03.095099 14173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:03.101425 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 14105
I20260619 16:44:03.101594 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:44:03.101903 392 external_mini_cluster.cc:1428] Setting key 90886a665b4232daac9490a28cb6d7d0
I20260619 16:44:03.102188 14173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331: Bootstrap starting.
I20260619 16:44:03.105139 14173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:03.106212 14173 log.cc:824] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331: Log is configured to *not* fsync() on all Append() calls
I20260619 16:44:03.108672 14173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331: No bootstrap required, opened a new log
Jun 19 16:44:03 dist-test-slave-03cd krb5kdc[14092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:44:03.113965 14173 raft_consensus.cc:348] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06d30f6f6d24164b9db02ecfed70331" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } }
I20260619 16:44:03.114236 14173 raft_consensus.cc:374] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:03.114306 14173 raft_consensus.cc:729] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e06d30f6f6d24164b9db02ecfed70331, State: Initialized, Role: FOLLOWER
I20260619 16:44:03.114869 14173 consensus_queue.cc:260] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06d30f6f6d24164b9db02ecfed70331" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } }
I20260619 16:44:03.115015 14173 raft_consensus.cc:388] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:44:03.115170 14173 raft_consensus.cc:482] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:44:03.115305 14173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:03.116387 14173 raft_consensus.cc:504] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06d30f6f6d24164b9db02ecfed70331" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } }
I20260619 16:44:03.116770 14173 leader_election.cc:302] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e06d30f6f6d24164b9db02ecfed70331; no voters:
I20260619 16:44:03.117159 14173 leader_election.cc:288] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:44:03.117236 14178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:44:03.117513 14178 raft_consensus.cc:686] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [term 1 LEADER]: Becoming Leader. State: Replica: e06d30f6f6d24164b9db02ecfed70331, State: Running, Role: LEADER
I20260619 16:44:03.117887 14178 consensus_queue.cc:237] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06d30f6f6d24164b9db02ecfed70331" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } }
I20260619 16:44:03.118323 14176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:03.103456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40432 (local address 127.0.98.62:34767)
0619 16:44:03.103999 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:03.104015 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:03.104051 (+ 36us) server_negotiation.cc:407] Connection header received
0619 16:44:03.104768 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:03.104786 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:03.105159 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:03.105514 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:03.106509 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.107507 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.108801 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.109332 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.109460 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:03.112877 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:03.112884 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:03.112903 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:03.112938 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:03.114960 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:03.115541 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:03.115547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:03.115552 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:03.115727 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:03.116027 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:03.116031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:03.116033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:03.116465 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:03.116725 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:03.117030 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 16:44:03.117383 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":224,"threads_started":1}
I20260619 16:44:03.118675 14173 sys_catalog.cc:564] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:44:03.119694 14179 sys_catalog.cc:455] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e06d30f6f6d24164b9db02ecfed70331" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06d30f6f6d24164b9db02ecfed70331" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } } }
I20260619 16:44:03.119935 14179 sys_catalog.cc:458] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [sys.catalog]: This master's current role is: LEADER
I20260619 16:44:03.119776 14178 sys_catalog.cc:455] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e06d30f6f6d24164b9db02ecfed70331. Latest consensus state: current_term: 1 leader_uuid: "e06d30f6f6d24164b9db02ecfed70331" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06d30f6f6d24164b9db02ecfed70331" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } } }
I20260619 16:44:03.120265 14178 sys_catalog.cc:458] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [sys.catalog]: This master's current role is: LEADER
I20260619 16:44:03.121482 14187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:44:03.124584 14187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:44:03.130741 14187 catalog_manager.cc:1347] Generated new cluster ID: 57073abe06e945c59849491335a9afb0
I20260619 16:44:03.130837 14187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:44:03.141763 14187 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:44:03.142822 14187 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:44:03.156839 14187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331: Generated new TSK 0
I20260619 16:44:03.157799 14187 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:44:03.195171 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34767
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:44:03.322316 14199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:03.322641 14199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:03.322691 14199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:03.327455 14199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:03.327562 14199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:03.327632 14199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:44:03.334338 14199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:44:03.335794 14199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:03.337136 14199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:03.347978 14204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:03.348026 14205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:03.348162 14199 server_base.cc:1034] running on GCE node
W20260619 16:44:03.347975 14207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:03.348770 14199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:03.349496 14199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:03.350697 14199 hybrid_clock.cc:648] HybridClock initialized: now 1781887443350682 us; error 27 us; skew 500 ppm
Jun 19 16:44:03 dist-test-slave-03cd krb5kdc[14092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:03.353971 14199 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:44:03.355057 14199 webserver.cc:460] Webserver started at http://127.0.98.1:36737/ using document root <none> and password file <none>
I20260619 16:44:03.355764 14199 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:03.355839 14199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:03.356091 14199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:44:03.358335 14199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/data/instance:
uuid: "c2e3597861584b1ba48bd52fe0489942"
format_stamp: "Formatted at 2026-06-19 16:44:03 on dist-test-slave-03cd"
server_key: "4bdc678131716693d680eb44e7b4744f"
server_key_iv: "3d90c0b223a987bebbadedf5e7ced098"
server_key_version: "encryptionkey@0"
I20260619 16:44:03.358961 14199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c2e3597861584b1ba48bd52fe0489942"
format_stamp: "Formatted at 2026-06-19 16:44:03 on dist-test-slave-03cd"
server_key: "4bdc678131716693d680eb44e7b4744f"
server_key_iv: "3d90c0b223a987bebbadedf5e7ced098"
server_key_version: "encryptionkey@0"
I20260619 16:44:03.363170 14199 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:44:03.365975 14214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:03.367401 14199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:44:03.367589 14199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "c2e3597861584b1ba48bd52fe0489942"
format_stamp: "Formatted at 2026-06-19 16:44:03 on dist-test-slave-03cd"
server_key: "4bdc678131716693d680eb44e7b4744f"
server_key_iv: "3d90c0b223a987bebbadedf5e7ced098"
server_key_version: "encryptionkey@0"
I20260619 16:44:03.367775 14199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:03.379850 14199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:03.384234 14199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:03.384472 14199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:03.385416 14199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:44:03.386554 14199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:44:03.386612 14199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:03.386659 14199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:44:03.386722 14199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:03.397948 14199 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45375
I20260619 16:44:03.397996 14327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45375 every 8 connection(s)
I20260619 16:44:03.399071 14199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:44:03.402922 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 14199
I20260619 16:44:03.403111 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:44:03.403383 392 external_mini_cluster.cc:1428] Setting key 61f64dab1b5b4cb9fcaac16ecd9e5e65
Jun 19 16:44:03 dist-test-slave-03cd krb5kdc[14092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 16:44:03.416965 14176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:03.401462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56137 (local address 127.0.98.62:34767)
0619 16:44:03.401660 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:03.401664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:03.402319 (+ 655us) server_negotiation.cc:407] Connection header received
0619 16:44:03.403951 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:03.403955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:03.404020 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:03.404131 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:03.405463 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.406019 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.408126 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.408544 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.408730 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:03.412220 (+ 3490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:03.412224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:03.412226 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:03.412257 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:03.413974 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:03.414608 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:03.414611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:03.414613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:03.414698 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:03.415244 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:03.415248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:03.415249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:03.415479 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:03.415565 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:03.416501 (+ 936us) server_negotiation.cc:299] Negotiation successful
0619 16:44:03.416697 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260619 16:44:03.417766 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:03.401576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:56137)
0619 16:44:03.402129 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:03.402180 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:03.403648 (+ 1468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:03.404330 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:03.404337 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:03.404751 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:03.405290 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:03.405304 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.406169 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.406173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:03.407605 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:03.407632 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.408817 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.408958 (+ 141us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:03.409281 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:03.410033 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:03.410053 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:03.411959 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:03.414176 (+ 2217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:03.414183 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:03.414199 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:03.414496 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:03.414901 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:03.414904 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:03.414906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:03.415090 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:03.415610 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:03.415617 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:03.416142 (+ 525us) client_negotiation.cc:769] Sending connection context
0619 16:44:03.416391 (+ 249us) client_negotiation.cc:240] Negotiation successful
0619 16:44:03.416728 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":139,"threads_started":1}
I20260619 16:44:03.419351 14328 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34767
I20260619 16:44:03.419726 14328 heartbeater.cc:464] Registering TS with master...
I20260619 16:44:03.420679 14328 heartbeater.cc:511] Master 127.0.98.62:34767 requested a full tablet report, sending...
I20260619 16:44:03.422585 14133 ts_manager.cc:194] Registered new tserver with Master: c2e3597861584b1ba48bd52fe0489942 (127.0.98.1:45375)
I20260619 16:44:03.423918 14133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:44:03.436380 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34767
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:44:03.554857 14333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:03.555159 14333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:03.555205 14333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:03.559712 14333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:03.559808 14333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:03.559911 14333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:44:03.565627 14333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:44:03.566833 14333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:03.567996 14333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:03.578222 14338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:03.578222 14341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:03.578289 14339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:03.578650 14333 server_base.cc:1034] running on GCE node
I20260619 16:44:03.579118 14333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:03.579880 14333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:03.581075 14333 hybrid_clock.cc:648] HybridClock initialized: now 1781887443581060 us; error 34 us; skew 500 ppm
Jun 19 16:44:03 dist-test-slave-03cd krb5kdc[14092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:03.584067 14333 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:44:03.585069 14333 webserver.cc:460] Webserver started at http://127.0.98.2:40345/ using document root <none> and password file <none>
I20260619 16:44:03.585639 14333 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:03.585698 14333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:03.585880 14333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:44:03.587999 14333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/data/instance:
uuid: "ae08116dded4430bb72b9c0a76b69d8c"
format_stamp: "Formatted at 2026-06-19 16:44:03 on dist-test-slave-03cd"
server_key: "cf0d6dec595e5e643a67363dffb9c331"
server_key_iv: "3a890b9b23bf8bc931b7c31fe937e7be"
server_key_version: "encryptionkey@0"
I20260619 16:44:03.588516 14333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ae08116dded4430bb72b9c0a76b69d8c"
format_stamp: "Formatted at 2026-06-19 16:44:03 on dist-test-slave-03cd"
server_key: "cf0d6dec595e5e643a67363dffb9c331"
server_key_iv: "3a890b9b23bf8bc931b7c31fe937e7be"
server_key_version: "encryptionkey@0"
I20260619 16:44:03.592374 14333 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:44:03.595105 14348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:03.596578 14333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:44:03.596704 14333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "ae08116dded4430bb72b9c0a76b69d8c"
format_stamp: "Formatted at 2026-06-19 16:44:03 on dist-test-slave-03cd"
server_key: "cf0d6dec595e5e643a67363dffb9c331"
server_key_iv: "3a890b9b23bf8bc931b7c31fe937e7be"
server_key_version: "encryptionkey@0"
I20260619 16:44:03.596896 14333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:03.613444 14333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:03.617102 14333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:03.617319 14333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:03.618079 14333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:44:03.619019 14333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:44:03.619077 14333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:03.619120 14333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:44:03.619135 14333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:03.629674 14333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43683
I20260619 16:44:03.629827 14461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43683 every 8 connection(s)
I20260619 16:44:03.631003 14333 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:44:03.633829 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 14333
I20260619 16:44:03.633956 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:44:03.634282 392 external_mini_cluster.cc:1428] Setting key e52747c67374744e104d1c17d593e91b
Jun 19 16:44:03 dist-test-slave-03cd krb5kdc[14092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 16:44:03.649437 14176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:03.633546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44589 (local address 127.0.98.62:34767)
0619 16:44:03.633722 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:03.633727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:03.635076 (+ 1349us) server_negotiation.cc:407] Connection header received
0619 16:44:03.637009 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:03.637013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:03.637067 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:03.637141 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:03.638800 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.639349 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.641298 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.641578 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.641697 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:03.644982 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:03.644986 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:03.644990 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:03.645020 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:03.646711 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:03.647400 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:03.647410 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:03.647412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:03.647492 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:03.647990 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:03.647993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:03.647995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:03.648279 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:03.648382 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:03.649177 (+ 795us) server_negotiation.cc:299] Negotiation successful
0619 16:44:03.649298 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 16:44:03.650588 14464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:03.634063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:44589)
0619 16:44:03.634911 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:03.634970 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:03.636770 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:03.637349 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:03.637360 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:03.637893 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:03.638626 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:03.638648 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.639481 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.639488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:03.641117 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:03.641131 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.641782 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.641790 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:03.642052 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:03.642783 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:03.642810 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:03.644751 (+ 1941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:03.646867 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:03.646874 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:03.646896 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:03.647248 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:03.647599 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:03.647603 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:03.647604 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:03.647879 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:03.648444 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:03.648450 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:03.648854 (+ 404us) client_negotiation.cc:769] Sending connection context
0619 16:44:03.649089 (+ 235us) client_negotiation.cc:240] Negotiation successful
0619 16:44:03.649498 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":157,"threads_started":1}
I20260619 16:44:03.651957 14462 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34767
I20260619 16:44:03.652242 14462 heartbeater.cc:464] Registering TS with master...
I20260619 16:44:03.652777 14462 heartbeater.cc:511] Master 127.0.98.62:34767 requested a full tablet report, sending...
I20260619 16:44:03.653966 14133 ts_manager.cc:194] Registered new tserver with Master: ae08116dded4430bb72b9c0a76b69d8c (127.0.98.2:43683)
I20260619 16:44:03.654570 14133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:44:03.668095 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34767
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:44:03.785719 14467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:03.786062 14467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:03.786114 14467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:03.791014 14467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:03.791116 14467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:03.791183 14467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:44:03.797288 14467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:44:03.798530 14467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:03.799695 14467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:03.810928 14472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:03.810937 14473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:03.811053 14467 server_base.cc:1034] running on GCE node
W20260619 16:44:03.810945 14475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:03.811830 14467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:03.812593 14467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:03.813820 14467 hybrid_clock.cc:648] HybridClock initialized: now 1781887443813776 us; error 58 us; skew 500 ppm
Jun 19 16:44:03 dist-test-slave-03cd krb5kdc[14092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:03.816901 14467 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:44:03.818101 14467 webserver.cc:460] Webserver started at http://127.0.98.3:45009/ using document root <none> and password file <none>
I20260619 16:44:03.818745 14467 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:03.818861 14467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:03.819152 14467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:44:03.821832 14467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/data/instance:
uuid: "6791ec79dd4d44b084a2221643274df6"
format_stamp: "Formatted at 2026-06-19 16:44:03 on dist-test-slave-03cd"
server_key: "4a84e24a3cf93c41985099ecf5eb1063"
server_key_iv: "56c2e3f95d6b3cebca369d78f98e9b64"
server_key_version: "encryptionkey@0"
I20260619 16:44:03.822507 14467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6791ec79dd4d44b084a2221643274df6"
format_stamp: "Formatted at 2026-06-19 16:44:03 on dist-test-slave-03cd"
server_key: "4a84e24a3cf93c41985099ecf5eb1063"
server_key_iv: "56c2e3f95d6b3cebca369d78f98e9b64"
server_key_version: "encryptionkey@0"
I20260619 16:44:03.826865 14467 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260619 16:44:03.829968 14482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:03.831490 14467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 16:44:03.831715 14467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "6791ec79dd4d44b084a2221643274df6"
format_stamp: "Formatted at 2026-06-19 16:44:03 on dist-test-slave-03cd"
server_key: "4a84e24a3cf93c41985099ecf5eb1063"
server_key_iv: "56c2e3f95d6b3cebca369d78f98e9b64"
server_key_version: "encryptionkey@0"
I20260619 16:44:03.831879 14467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:03.842859 14467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:03.846676 14467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:03.846949 14467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:03.847625 14467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:44:03.848673 14467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:44:03.848750 14467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:03.848812 14467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:44:03.848857 14467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:03.859229 14467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45541
I20260619 16:44:03.859248 14595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45541 every 8 connection(s)
I20260619 16:44:03.860630 14467 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:44:03.865429 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 14467
I20260619 16:44:03.865583 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:44:03.865971 392 external_mini_cluster.cc:1428] Setting key 60aec86016d3166bb27ab3c6dfc13a49
Jun 19 16:44:03 dist-test-slave-03cd krb5kdc[14092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:44:03.877269 14176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:03.862844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49817 (local address 127.0.98.62:34767)
0619 16:44:03.863214 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:03.863218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:03.864231 (+ 1013us) server_negotiation.cc:407] Connection header received
0619 16:44:03.865826 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:03.865830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:03.865900 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:03.865997 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:03.867681 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.868399 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.869757 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.869984 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.870115 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:03.873158 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:03.873162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:03.873179 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:03.873209 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:03.874738 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:03.875348 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:03.875351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:03.875353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:03.875442 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:03.875986 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:03.875990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:03.875991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:03.876230 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:03.876329 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:03.876979 (+ 650us) server_negotiation.cc:299] Negotiation successful
0619 16:44:03.877112 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 16:44:03.878175 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:03.863297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:49817)
0619 16:44:03.864021 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:03.864084 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:03.865458 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:03.866217 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:03.866226 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:03.866752 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:03.867459 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:03.867485 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.868564 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.868568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:03.869568 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:03.869579 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.870182 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.870186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:03.870361 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:03.871093 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:03.871110 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:03.872931 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:03.874886 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:03.874892 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:03.874910 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:03.875227 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:03.875566 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:03.875570 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:03.875571 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:03.875809 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:03.876370 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:03.876376 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:03.876675 (+ 299us) client_negotiation.cc:769] Sending connection context
0619 16:44:03.876854 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 16:44:03.877195 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":210,"threads_started":1}
I20260619 16:44:03.879732 14596 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34767
I20260619 16:44:03.880045 14596 heartbeater.cc:464] Registering TS with master...
I20260619 16:44:03.880685 14596 heartbeater.cc:511] Master 127.0.98.62:34767 requested a full tablet report, sending...
I20260619 16:44:03.881819 14133 ts_manager.cc:194] Registered new tserver with Master: 6791ec79dd4d44b084a2221643274df6 (127.0.98.3:45541)
I20260619 16:44:03.882495 14133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49817
I20260619 16:44:03.893656 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:44:03.905107 14176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:03.896100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40436 (local address 127.0.98.62:34767)
0619 16:44:03.896264 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:03.896271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:03.896536 (+ 265us) server_negotiation.cc:407] Connection header received
0619 16:44:03.896591 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:03.896593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:03.896684 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:03.896772 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:03.897534 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.898200 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.899095 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.899319 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.899442 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:03.901320 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:03.901323 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:03.901326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:03.901361 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:03.903160 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:03.903648 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:03.903651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:03.903653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:03.903826 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:03.904131 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:03.904134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:03.904135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:03.904386 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:03.904493 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:03.904720 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 16:44:03.904833 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:44:03.909197 14133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:44:03.912438 14133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:44:03.928937 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:03.922761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45375 (local address 127.0.0.1:54316)
0619 16:44:03.923613 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:03.923699 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:03.923830 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:03.924520 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:03.924526 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:03.924552 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:03.924706 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:03.924714 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.925946 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.925949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:03.927361 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:03.927371 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.928346 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.928350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:03.928456 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:03.928492 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 16:44:03.928615 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 16:44:03.928758 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":103,"threads_started":1}
I20260619 16:44:03.928937 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:03.922773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43683 (local address 127.0.0.1:45966)
0619 16:44:03.924430 (+ 1657us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:03.924445 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:03.924543 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:03.924971 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:03.924973 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:03.924990 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:03.925078 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:03.925084 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.926339 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.926343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:03.927441 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:03.927451 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.928584 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.928590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:03.928677 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:03.928696 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 16:44:03.928753 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 16:44:03.928804 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1596,"mutex_wait_us":1,"thread_start_us":74,"threads_started":1}
I20260619 16:44:03.929046 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:03.923065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45966 (local address 127.0.98.2:43683)
0619 16:44:03.923575 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:03.923586 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:03.924558 (+ 972us) server_negotiation.cc:407] Connection header received
0619 16:44:03.924643 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:03.924651 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:03.924828 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:03.924976 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:03.925163 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.926192 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.927601 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.928414 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.928573 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:03.928722 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:03.928781 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 16:44:03.928881 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":141,"threads_started":1}
I20260619 16:44:03.929705 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:03.923501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45541 (local address 127.0.0.1:48750)
0619 16:44:03.924705 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:03.924717 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:03.924806 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:03.925381 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:03.925384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:03.925404 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:03.925495 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:03.925501 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.926777 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.926784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:03.927684 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:03.927694 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.928842 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.928846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:03.928932 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:03.928955 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:44:03.929049 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 16:44:03.929570 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1150,"thread_start_us":99,"threads_started":1}
I20260619 16:44:03.929803 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:03.923974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48750 (local address 127.0.98.3:45541)
0619 16:44:03.924362 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:03.924373 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:03.924817 (+ 444us) server_negotiation.cc:407] Connection header received
0619 16:44:03.924902 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:03.924908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:03.925075 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:03.925220 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:03.925647 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.926659 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.927790 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.928574 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.928671 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:03.928782 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:03.929540 (+ 758us) server_negotiation.cc:299] Negotiation successful
0619 16:44:03.929651 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":195,"threads_started":1}
I20260619 16:44:03.930336 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:03.923822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54316 (local address 127.0.98.1:45375)
0619 16:44:03.924115 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:03.924119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:03.924139 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:44:03.924199 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:03.924203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:03.924371 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:03.924493 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:03.924846 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.925822 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.927531 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.928217 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.928349 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:03.928507 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:03.930065 (+ 1558us) server_negotiation.cc:299] Negotiation successful
0619 16:44:03.930191 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":116,"threads_started":1}
I20260619 16:44:03.932643 14396 tablet_service.cc:1467] Processing CreateTablet for tablet e9ed4b1ff2d847b2a083e4f2c1aa32bd (DEFAULT_TABLE table=test-table [id=2b72e7404c48403d86f3ebff68d54dd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:44:03.932641 14530 tablet_service.cc:1467] Processing CreateTablet for tablet e9ed4b1ff2d847b2a083e4f2c1aa32bd (DEFAULT_TABLE table=test-table [id=2b72e7404c48403d86f3ebff68d54dd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:44:03.933905 14396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9ed4b1ff2d847b2a083e4f2c1aa32bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:03.933908 14530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9ed4b1ff2d847b2a083e4f2c1aa32bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:03.935377 14262 tablet_service.cc:1467] Processing CreateTablet for tablet e9ed4b1ff2d847b2a083e4f2c1aa32bd (DEFAULT_TABLE table=test-table [id=2b72e7404c48403d86f3ebff68d54dd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:44:03.936494 14262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9ed4b1ff2d847b2a083e4f2c1aa32bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:03.941901 14613 tablet_bootstrap.cc:492] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6: Bootstrap starting.
I20260619 16:44:03.942358 14614 tablet_bootstrap.cc:492] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P ae08116dded4430bb72b9c0a76b69d8c: Bootstrap starting.
I20260619 16:44:03.943404 14615 tablet_bootstrap.cc:492] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P c2e3597861584b1ba48bd52fe0489942: Bootstrap starting.
I20260619 16:44:03.944927 14613 tablet_bootstrap.cc:654] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:03.945387 14614 tablet_bootstrap.cc:654] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P ae08116dded4430bb72b9c0a76b69d8c: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:03.946061 14615 tablet_bootstrap.cc:654] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P c2e3597861584b1ba48bd52fe0489942: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:03.946177 14614 log.cc:824] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P ae08116dded4430bb72b9c0a76b69d8c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:44:03.946215 14613 log.cc:824] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6: Log is configured to *not* fsync() on all Append() calls
I20260619 16:44:03.946915 14615 log.cc:824] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P c2e3597861584b1ba48bd52fe0489942: Log is configured to *not* fsync() on all Append() calls
I20260619 16:44:03.948313 14614 tablet_bootstrap.cc:492] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P ae08116dded4430bb72b9c0a76b69d8c: No bootstrap required, opened a new log
I20260619 16:44:03.948448 14613 tablet_bootstrap.cc:492] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6: No bootstrap required, opened a new log
I20260619 16:44:03.948565 14614 ts_tablet_manager.cc:1397] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P ae08116dded4430bb72b9c0a76b69d8c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 16:44:03.948652 14613 ts_tablet_manager.cc:1397] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260619 16:44:03.949002 14615 tablet_bootstrap.cc:492] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P c2e3597861584b1ba48bd52fe0489942: No bootstrap required, opened a new log
I20260619 16:44:03.949187 14615 ts_tablet_manager.cc:1397] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P c2e3597861584b1ba48bd52fe0489942: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 16:44:03.951947 14614 raft_consensus.cc:348] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P ae08116dded4430bb72b9c0a76b69d8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 } } peers { permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 } } peers { permanent_uuid: "6791ec79dd4d44b084a2221643274df6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45541 } }
I20260619 16:44:03.952284 14614 raft_consensus.cc:374] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P ae08116dded4430bb72b9c0a76b69d8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:03.952342 14614 raft_consensus.cc:729] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P ae08116dded4430bb72b9c0a76b69d8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae08116dded4430bb72b9c0a76b69d8c, State: Initialized, Role: FOLLOWER
I20260619 16:44:03.952790 14614 consensus_queue.cc:260] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P ae08116dded4430bb72b9c0a76b69d8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 } } peers { permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 } } peers { permanent_uuid: "6791ec79dd4d44b084a2221643274df6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45541 } }
I20260619 16:44:03.952905 14613 raft_consensus.cc:348] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 } } peers { permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 } } peers { permanent_uuid: "6791ec79dd4d44b084a2221643274df6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45541 } }
I20260619 16:44:03.953198 14613 raft_consensus.cc:374] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:03.953267 14613 raft_consensus.cc:729] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6791ec79dd4d44b084a2221643274df6, State: Initialized, Role: FOLLOWER
I20260619 16:44:03.953709 14613 consensus_queue.cc:260] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 } } peers { permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 } } peers { permanent_uuid: "6791ec79dd4d44b084a2221643274df6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45541 } }
I20260619 16:44:03.953869 14614 ts_tablet_manager.cc:1428] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P ae08116dded4430bb72b9c0a76b69d8c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 16:44:03.953754 14615 raft_consensus.cc:348] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P c2e3597861584b1ba48bd52fe0489942 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 } } peers { permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 } } peers { permanent_uuid: "6791ec79dd4d44b084a2221643274df6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45541 } }
I20260619 16:44:03.954034 14615 raft_consensus.cc:374] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P c2e3597861584b1ba48bd52fe0489942 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:03.954106 14615 raft_consensus.cc:729] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P c2e3597861584b1ba48bd52fe0489942 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2e3597861584b1ba48bd52fe0489942, State: Initialized, Role: FOLLOWER
I20260619 16:44:03.954277 14462 heartbeater.cc:503] Master 127.0.98.62:34767 was elected leader, sending a full tablet report...
I20260619 16:44:03.954555 14596 heartbeater.cc:503] Master 127.0.98.62:34767 was elected leader, sending a full tablet report...
I20260619 16:44:03.954734 14613 ts_tablet_manager.cc:1428] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260619 16:44:03.955564 14615 consensus_queue.cc:260] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P c2e3597861584b1ba48bd52fe0489942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 } } peers { permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 } } peers { permanent_uuid: "6791ec79dd4d44b084a2221643274df6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45541 } }
I20260619 16:44:03.956279 14328 heartbeater.cc:503] Master 127.0.98.62:34767 was elected leader, sending a full tablet report...
I20260619 16:44:03.956586 14615 ts_tablet_manager.cc:1428] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P c2e3597861584b1ba48bd52fe0489942: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260619 16:44:04.111846 14597 tablet.cc:2369] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:44:04.128530 14620 raft_consensus.cc:482] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:44:04.128844 14620 raft_consensus.cc:504] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 } } peers { permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 } } peers { permanent_uuid: "6791ec79dd4d44b084a2221643274df6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45541 } }
I20260619 16:44:04.130105 14620 leader_election.cc:288] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2e3597861584b1ba48bd52fe0489942 (127.0.98.1:45375), ae08116dded4430bb72b9c0a76b69d8c (127.0.98.2:43683)
W20260619 16:44:04.132483 14463 tablet.cc:2369] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P ae08116dded4430bb72b9c0a76b69d8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:44:04.134529 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.130469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43683 (local address 127.0.98.3:43709)
0619 16:44:04.130614 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:04.130631 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:04.130747 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:04.131106 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:04.131109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:04.131141 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:04.131275 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.131286 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.132149 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.132152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.133127 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.133138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.134036 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.134040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.134154 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.134182 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 16:44:04.134274 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 16:44:04.134357 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 16:44:04.134582 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.130602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43709 (local address 127.0.98.2:43683)
0619 16:44:04.130729 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:04.130733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:04.130749 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:44:04.130894 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:04.130898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:04.130959 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:04.131038 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:04.131399 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.132013 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.133304 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.133883 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.133978 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.134035 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:04.134317 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 16:44:04.134397 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:44:04.135068 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.130469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45375 (local address 127.0.98.3:34579)
0619 16:44:04.130922 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:04.130934 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:04.131030 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:04.131323 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:04.131325 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:04.131362 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:04.131475 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.131482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.132333 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.132337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.133613 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.133624 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.134614 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.134619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.134721 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.134755 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 16:44:04.134849 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 16:44:04.134921 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":129,"thread_start_us":141,"threads_started":1}
I20260619 16:44:04.135126 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.130709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34579 (local address 127.0.98.1:45375)
0619 16:44:04.130851 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:04.130856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:04.131104 (+ 248us) server_negotiation.cc:407] Connection header received
0619 16:44:04.131162 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:04.131164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:04.131214 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:04.131297 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:04.131596 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.132193 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.133763 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.134424 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.134530 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.134610 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:04.134869 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 16:44:04.135004 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:44:04.135371 14416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ed4b1ff2d847b2a083e4f2c1aa32bd" candidate_uuid: "6791ec79dd4d44b084a2221643274df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae08116dded4430bb72b9c0a76b69d8c" is_pre_election: true
I20260619 16:44:04.135887 14416 raft_consensus.cc:2454] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P ae08116dded4430bb72b9c0a76b69d8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6791ec79dd4d44b084a2221643274df6 in term 0.
I20260619 16:44:04.135840 14282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ed4b1ff2d847b2a083e4f2c1aa32bd" candidate_uuid: "6791ec79dd4d44b084a2221643274df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2e3597861584b1ba48bd52fe0489942" is_pre_election: true
I20260619 16:44:04.136133 14282 raft_consensus.cc:2454] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P c2e3597861584b1ba48bd52fe0489942 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6791ec79dd4d44b084a2221643274df6 in term 0.
I20260619 16:44:04.136502 14484 leader_election.cc:302] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6791ec79dd4d44b084a2221643274df6, ae08116dded4430bb72b9c0a76b69d8c; no voters:
I20260619 16:44:04.136848 14620 raft_consensus.cc:2781] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:44:04.136955 14620 raft_consensus.cc:482] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:44:04.137038 14620 raft_consensus.cc:3037] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:04.138151 14620 raft_consensus.cc:504] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 } } peers { permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 } } peers { permanent_uuid: "6791ec79dd4d44b084a2221643274df6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45541 } }
I20260619 16:44:04.138549 14620 leader_election.cc:288] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6 [CANDIDATE]: Term 1 election: Requested vote from peers c2e3597861584b1ba48bd52fe0489942 (127.0.98.1:45375), ae08116dded4430bb72b9c0a76b69d8c (127.0.98.2:43683)
I20260619 16:44:04.139070 14416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ed4b1ff2d847b2a083e4f2c1aa32bd" candidate_uuid: "6791ec79dd4d44b084a2221643274df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae08116dded4430bb72b9c0a76b69d8c"
I20260619 16:44:04.139070 14282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ed4b1ff2d847b2a083e4f2c1aa32bd" candidate_uuid: "6791ec79dd4d44b084a2221643274df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2e3597861584b1ba48bd52fe0489942"
I20260619 16:44:04.139219 14416 raft_consensus.cc:3037] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P ae08116dded4430bb72b9c0a76b69d8c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:04.139219 14282 raft_consensus.cc:3037] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P c2e3597861584b1ba48bd52fe0489942 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:04.140704 14416 raft_consensus.cc:2454] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P ae08116dded4430bb72b9c0a76b69d8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6791ec79dd4d44b084a2221643274df6 in term 1.
I20260619 16:44:04.140704 14282 raft_consensus.cc:2454] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P c2e3597861584b1ba48bd52fe0489942 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6791ec79dd4d44b084a2221643274df6 in term 1.
I20260619 16:44:04.141096 14484 leader_election.cc:302] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6791ec79dd4d44b084a2221643274df6, ae08116dded4430bb72b9c0a76b69d8c; no voters:
I20260619 16:44:04.141394 14620 raft_consensus.cc:2781] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:44:04.141666 14620 raft_consensus.cc:686] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6 [term 1 LEADER]: Becoming Leader. State: Replica: 6791ec79dd4d44b084a2221643274df6, State: Running, Role: LEADER
I20260619 16:44:04.142028 14620 consensus_queue.cc:237] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 } } peers { permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 } } peers { permanent_uuid: "6791ec79dd4d44b084a2221643274df6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45541 } }
I20260619 16:44:04.145676 14136 catalog_manager.cc:5495] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6791ec79dd4d44b084a2221643274df6 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6791ec79dd4d44b084a2221643274df6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6791ec79dd4d44b084a2221643274df6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45541 } health_report { overall_health: HEALTHY } } }
W20260619 16:44:04.150689 14329 tablet.cc:2369] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P c2e3597861584b1ba48bd52fe0489942: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:44:04.169015 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.165260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48762 (local address 127.0.98.3:45541)
0619 16:44:04.165411 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:04.165416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:04.165432 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:44:04.165567 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:04.165570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:04.165627 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:04.165745 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:44:04.166044 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.166587 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.167460 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.167739 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.167848 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.168178 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:44:04.168624 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:44:04.168763 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:04.168804 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 16:44:04.168864 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:44:04.175383 14416 raft_consensus.cc:1261] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P ae08116dded4430bb72b9c0a76b69d8c [term 1 FOLLOWER]: Refusing update from remote peer 6791ec79dd4d44b084a2221643274df6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:44:04.175381 14282 raft_consensus.cc:1261] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P c2e3597861584b1ba48bd52fe0489942 [term 1 FOLLOWER]: Refusing update from remote peer 6791ec79dd4d44b084a2221643274df6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:44:04.176282 14623 consensus_queue.cc:1035] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:44:04.176497 14620 consensus_queue.cc:1035] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:44:04.196739 14133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:44:04.196929 14133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:44:04.199815 14133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331: Sending DeleteTablet for 3 replicas of tablet e9ed4b1ff2d847b2a083e4f2c1aa32bd
I20260619 16:44:04.200747 14262 tablet_service.cc:1514] Processing DeleteTablet for tablet e9ed4b1ff2d847b2a083e4f2c1aa32bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54316
I20260619 16:44:04.200750 14396 tablet_service.cc:1514] Processing DeleteTablet for tablet e9ed4b1ff2d847b2a083e4f2c1aa32bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45966
I20260619 16:44:04.201131 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 14199
I20260619 16:44:04.201458 14637 tablet_replica.cc:330] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P ae08116dded4430bb72b9c0a76b69d8c: stopping tablet replica
I20260619 16:44:04.202091 14637 raft_consensus.cc:2229] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P ae08116dded4430bb72b9c0a76b69d8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:44:04.202587 14637 raft_consensus.cc:2258] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P ae08116dded4430bb72b9c0a76b69d8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:44:04.202612 14530 tablet_service.cc:1514] Processing DeleteTablet for tablet e9ed4b1ff2d847b2a083e4f2c1aa32bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48750
I20260619 16:44:04.204537 14637 ts_tablet_manager.cc:1905] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P ae08116dded4430bb72b9c0a76b69d8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:44:04.207281 14638 tablet_replica.cc:330] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6: stopping tablet replica
I20260619 16:44:04.207649 14638 raft_consensus.cc:2229] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:44:04.208140 14638 raft_consensus.cc:2258] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:44:04.208595 14637 ts_tablet_manager.cc:1918] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P ae08116dded4430bb72b9c0a76b69d8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:44:04.208758 14637 log.cc:1196] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P ae08116dded4430bb72b9c0a76b69d8c: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/wal/wals/e9ed4b1ff2d847b2a083e4f2c1aa32bd
I20260619 16:44:04.209199 14637 ts_tablet_manager.cc:1939] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P ae08116dded4430bb72b9c0a76b69d8c: Deleting consensus metadata
I20260619 16:44:04.209724 14638 ts_tablet_manager.cc:1905] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 16:44:04.209857 14123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45375 (error 108)
I20260619 16:44:04.210704 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 14333
I20260619 16:44:04.210785 14122 catalog_manager.cc:4841] TS ae08116dded4430bb72b9c0a76b69d8c (127.0.98.2:43683): tablet e9ed4b1ff2d847b2a083e4f2c1aa32bd (table test-table [id=2b72e7404c48403d86f3ebff68d54dd0]) successfully deleted
I20260619 16:44:04.212960 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.212559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45375 (local address 127.0.0.1:38368)
0619 16:44:04.212773 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:04.212853 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260619 16:44:04.213153 14123 catalog_manager.cc:4584] TS c2e3597861584b1ba48bd52fe0489942 (127.0.98.1:45375): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e9ed4b1ff2d847b2a083e4f2c1aa32bd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45375: connect: Connection refused (error 111)
I20260619 16:44:04.213790 14638 ts_tablet_manager.cc:1918] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:44:04.213898 14638 log.cc:1196] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/wal/wals/e9ed4b1ff2d847b2a083e4f2c1aa32bd
I20260619 16:44:04.214293 14638 ts_tablet_manager.cc:1939] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P 6791ec79dd4d44b084a2221643274df6: Deleting consensus metadata
I20260619 16:44:04.215535 14123 catalog_manager.cc:4841] TS 6791ec79dd4d44b084a2221643274df6 (127.0.98.3:45541): tablet e9ed4b1ff2d847b2a083e4f2c1aa32bd (table test-table [id=2b72e7404c48403d86f3ebff68d54dd0]) successfully deleted
I20260619 16:44:04.217420 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 14467
I20260619 16:44:04.223738 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 14105
I20260619 16:44:04.231356 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34767
--webserver_interface=127.0.98.62
--webserver_port=37499
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:44:04.347707 14640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:04.348006 14640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:04.348054 14640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:04.352350 14640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:44:04.352432 14640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:04.352453 14640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:04.352471 14640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:44:04.352486 14640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:44:04.358363 14640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:44:04.359474 14640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:04.360579 14640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:04.374658 14649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:04.374655 14647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:04.374670 14646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:04.374752 14640 server_base.cc:1034] running on GCE node
I20260619 16:44:04.375766 14640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:04.377038 14640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:04.378226 14640 hybrid_clock.cc:648] HybridClock initialized: now 1781887444378198 us; error 45 us; skew 500 ppm
Jun 19 16:44:04 dist-test-slave-03cd krb5kdc[14092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:04.381443 14640 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:44:04.382573 14640 webserver.cc:460] Webserver started at http://127.0.98.62:37499/ using document root <none> and password file <none>
I20260619 16:44:04.383226 14640 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:04.383353 14640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:04.387902 14640 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:44:04.390496 14656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:04.392202 14640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:44:04.392398 14640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "e06d30f6f6d24164b9db02ecfed70331"
format_stamp: "Formatted at 2026-06-19 16:44:03 on dist-test-slave-03cd"
server_key: "baa2404c716818f086beba88a69cfdfa"
server_key_iv: "7d884047083ab07b9f332bf4a7b5519f"
server_key_version: "encryptionkey@0"
I20260619 16:44:04.393105 14640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:04.418681 14640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:04.422641 14640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:04.422884 14640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:04.430610 14640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34767
I20260619 16:44:04.430979 14708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34767 every 8 connection(s)
I20260619 16:44:04.431866 14640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:44:04.436599 14709 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:44:04.438529 14709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331: Bootstrap starting.
I20260619 16:44:04.439623 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 14640
I20260619 16:44:04.440137 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45375
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34767
--webserver_port=36737
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:44:04.457192 14709 log.cc:824] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331: Log is configured to *not* fsync() on all Append() calls
I20260619 16:44:04.466373 14709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:44:04.466892 14709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331: Bootstrap complete.
I20260619 16:44:04.471100 14709 raft_consensus.cc:348] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06d30f6f6d24164b9db02ecfed70331" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } }
I20260619 16:44:04.471522 14709 raft_consensus.cc:729] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e06d30f6f6d24164b9db02ecfed70331, State: Initialized, Role: FOLLOWER
I20260619 16:44:04.472062 14709 consensus_queue.cc:260] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06d30f6f6d24164b9db02ecfed70331" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } }
I20260619 16:44:04.472183 14709 raft_consensus.cc:388] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:44:04.472283 14709 raft_consensus.cc:482] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:44:04.472404 14709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:44:04.474509 14709 raft_consensus.cc:504] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06d30f6f6d24164b9db02ecfed70331" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } }
I20260619 16:44:04.474862 14709 leader_election.cc:302] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e06d30f6f6d24164b9db02ecfed70331; no voters:
I20260619 16:44:04.475205 14709 leader_election.cc:288] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 16:44:04.475297 14713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:44:04.475513 14713 raft_consensus.cc:686] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [term 2 LEADER]: Becoming Leader. State: Replica: e06d30f6f6d24164b9db02ecfed70331, State: Running, Role: LEADER
I20260619 16:44:04.475839 14713 consensus_queue.cc:237] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06d30f6f6d24164b9db02ecfed70331" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } }
I20260619 16:44:04.476518 14709 sys_catalog.cc:564] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:44:04.477339 14714 sys_catalog.cc:455] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e06d30f6f6d24164b9db02ecfed70331" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06d30f6f6d24164b9db02ecfed70331" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } } }
I20260619 16:44:04.477353 14715 sys_catalog.cc:455] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e06d30f6f6d24164b9db02ecfed70331. Latest consensus state: current_term: 2 leader_uuid: "e06d30f6f6d24164b9db02ecfed70331" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06d30f6f6d24164b9db02ecfed70331" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } } }
I20260619 16:44:04.477475 14714 sys_catalog.cc:458] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [sys.catalog]: This master's current role is: LEADER
I20260619 16:44:04.477487 14715 sys_catalog.cc:458] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [sys.catalog]: This master's current role is: LEADER
I20260619 16:44:04.478652 14722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:44:04.483620 14722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:44:04.484302 14722 catalog_manager.cc:1259] Loaded cluster ID: 57073abe06e945c59849491335a9afb0
I20260619 16:44:04.484377 14722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:44:04.485807 14722 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 16:44:04.490401 14722 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 16:44:04.496503 14722 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:44:04.497474 14722 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331: Loaded TSK: 0
I20260619 16:44:04.498724 14722 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:44:04.562425 14711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:04.562774 14711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:04.562860 14711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:04.567215 14711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:04.567309 14711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:04.567400 14711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:44:04.573055 14711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36737
--tserver_master_addrs=127.0.98.62:34767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:44:04.574213 14711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:04.575271 14711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:04.585443 14736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:04.585486 14737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:04.585558 14739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:04.585942 14711 server_base.cc:1034] running on GCE node
I20260619 16:44:04.586416 14711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:04.587302 14711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:04.588503 14711 hybrid_clock.cc:648] HybridClock initialized: now 1781887444588468 us; error 48 us; skew 500 ppm
Jun 19 16:44:04 dist-test-slave-03cd krb5kdc[14092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:04.591921 14711 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:44:04.592890 14711 webserver.cc:460] Webserver started at http://127.0.98.1:36737/ using document root <none> and password file <none>
I20260619 16:44:04.593499 14711 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:04.593575 14711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:04.598220 14711 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:44:04.600665 14746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:04.602033 14711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:44:04.602181 14711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "c2e3597861584b1ba48bd52fe0489942"
format_stamp: "Formatted at 2026-06-19 16:44:03 on dist-test-slave-03cd"
server_key: "4bdc678131716693d680eb44e7b4744f"
server_key_iv: "3d90c0b223a987bebbadedf5e7ced098"
server_key_version: "encryptionkey@0"
I20260619 16:44:04.602747 14711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:04.628414 14711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:04.631981 14711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:04.632247 14711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:04.632889 14711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:44:04.634202 14753 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 16:44:04.638444 14711 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 16:44:04.638543 14711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260619 16:44:04.638653 14711 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 16:44:04.640493 14711 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 16:44:04.640571 14711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260619 16:44:04.640957 14753 tablet_bootstrap.cc:492] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P c2e3597861584b1ba48bd52fe0489942: Bootstrap starting.
I20260619 16:44:04.651087 14711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45375
I20260619 16:44:04.651187 14860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45375 every 8 connection(s)
I20260619 16:44:04.652201 14711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:44:04.658162 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 14711
I20260619 16:44:04.658706 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43683
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34767
--webserver_port=40345
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:44:04.661764 14753 log.cc:824] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P c2e3597861584b1ba48bd52fe0489942: Log is configured to *not* fsync() on all Append() calls
Jun 19 16:44:04 dist-test-slave-03cd krb5kdc[14092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:44:04.671399 14864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.654208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:60141)
0619 16:44:04.654831 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:04.654884 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:04.655772 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:04.657237 (+ 1465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:04.657244 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:04.657614 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:04.658075 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.658090 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.659636 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.659639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.660955 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.660968 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.662069 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.662076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.662312 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.663884 (+ 1572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:04.663912 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:04.665614 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:04.667987 (+ 2373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:04.667993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:04.668006 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:04.668281 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:04.668667 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:04.668671 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:04.668673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:04.668825 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:04.669637 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:04.669644 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:04.669911 (+ 267us) client_negotiation.cc:769] Sending connection context
0619 16:44:04.670097 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 16:44:04.670459 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":133,"threads_started":1}
I20260619 16:44:04.671386 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.654296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60141 (local address 127.0.98.62:34767)
0619 16:44:04.654749 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:04.654774 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:04.655026 (+ 252us) server_negotiation.cc:407] Connection header received
0619 16:44:04.656281 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:04.656301 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:04.656767 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:04.657178 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:04.658278 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.659445 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.661112 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.661833 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.662042 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.665882 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:04.665887 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:04.665900 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:04.665932 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:04.667843 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:04.668426 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:04.668431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:04.668435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:04.668563 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:04.668935 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:04.668944 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:04.668946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:04.669514 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:04.669666 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:04.670223 (+ 557us) server_negotiation.cc:299] Negotiation successful
0619 16:44:04.670500 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":123,"threads_started":1}
I20260619 16:44:04.672403 14753 tablet_bootstrap.cc:492] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P c2e3597861584b1ba48bd52fe0489942: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:44:04.672844 14753 tablet_bootstrap.cc:492] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P c2e3597861584b1ba48bd52fe0489942: Bootstrap complete.
I20260619 16:44:04.673902 14861 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34767
I20260619 16:44:04.674145 14861 heartbeater.cc:464] Registering TS with master...
I20260619 16:44:04.674309 14753 ts_tablet_manager.cc:1397] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P c2e3597861584b1ba48bd52fe0489942: Time spent bootstrapping tablet: real 0.034s user 0.026s sys 0.004s
I20260619 16:44:04.674723 14861 heartbeater.cc:511] Master 127.0.98.62:34767 requested a full tablet report, sending...
I20260619 16:44:04.677155 14671 ts_manager.cc:194] Registered new tserver with Master: c2e3597861584b1ba48bd52fe0489942 (127.0.98.1:45375)
I20260619 16:44:04.677381 14753 raft_consensus.cc:348] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P c2e3597861584b1ba48bd52fe0489942 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 } } peers { permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 } } peers { permanent_uuid: "6791ec79dd4d44b084a2221643274df6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45541 } }
I20260619 16:44:04.677913 14753 raft_consensus.cc:729] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P c2e3597861584b1ba48bd52fe0489942 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2e3597861584b1ba48bd52fe0489942, State: Initialized, Role: FOLLOWER
I20260619 16:44:04.678412 14753 consensus_queue.cc:260] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P c2e3597861584b1ba48bd52fe0489942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 } } peers { permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 } } peers { permanent_uuid: "6791ec79dd4d44b084a2221643274df6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45541 } }
I20260619 16:44:04.679533 14753 ts_tablet_manager.cc:1428] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P c2e3597861584b1ba48bd52fe0489942: Time spent starting tablet: real 0.005s user 0.003s sys 0.004s
I20260619 16:44:04.680909 14671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60141
I20260619 16:44:04.683506 14861 heartbeater.cc:503] Master 127.0.98.62:34767 was elected leader, sending a full tablet report...
Jun 19 16:44:04 dist-test-slave-03cd krb5kdc[14092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:44:04.693266 14869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.680532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38378 (local address 127.0.98.1:45375)
0619 16:44:04.681242 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:04.681259 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:04.681810 (+ 551us) server_negotiation.cc:407] Connection header received
0619 16:44:04.681881 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:04.681887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:04.682095 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:04.682225 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:04.683012 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.684201 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.685658 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.686066 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.686163 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.688929 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:04.688941 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:04.688952 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:04.688977 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:04.690933 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:04.691622 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:04.691628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:04.691630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:04.691781 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:04.692133 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:04.692138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:04.692140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:04.692421 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:04.692565 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:04.692921 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 16:44:04.693082 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":99,"threads_started":1}
I20260619 16:44:04.693266 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.680435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45375 (local address 127.0.0.1:38378)
0619 16:44:04.681498 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:04.681546 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:04.681684 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:04.682266 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:04.682273 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:04.682727 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:04.682873 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.682882 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.684346 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.684350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.685492 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.685507 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.686186 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.686209 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.686336 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.687056 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:04.687071 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:04.688735 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:04.691096 (+ 2361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:04.691108 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:04.691114 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:04.691475 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:04.691933 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:04.691937 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:04.691939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:04.692036 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:04.692540 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:04.692546 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:04.692682 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 16:44:04.692833 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 16:44:04.693074 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"thread_start_us":112,"threads_started":1}
I20260619 16:44:04.694492 14795 tablet_service.cc:1514] Processing DeleteTablet for tablet e9ed4b1ff2d847b2a083e4f2c1aa32bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38378
I20260619 16:44:04.695611 14870 tablet_replica.cc:330] stopping tablet replica
I20260619 16:44:04.695971 14870 raft_consensus.cc:2229] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P c2e3597861584b1ba48bd52fe0489942 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:44:04.696240 14870 raft_consensus.cc:2258] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P c2e3597861584b1ba48bd52fe0489942 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:44:04.697368 14870 ts_tablet_manager.cc:1905] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P c2e3597861584b1ba48bd52fe0489942: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:44:04.701297 14870 ts_tablet_manager.cc:1918] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P c2e3597861584b1ba48bd52fe0489942: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:44:04.701448 14870 log.cc:1196] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P c2e3597861584b1ba48bd52fe0489942: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/wal/wals/e9ed4b1ff2d847b2a083e4f2c1aa32bd
I20260619 16:44:04.701725 14870 ts_tablet_manager.cc:1939] T e9ed4b1ff2d847b2a083e4f2c1aa32bd P c2e3597861584b1ba48bd52fe0489942: Deleting consensus metadata
I20260619 16:44:04.703682 14659 catalog_manager.cc:4841] TS c2e3597861584b1ba48bd52fe0489942 (127.0.98.1:45375): tablet e9ed4b1ff2d847b2a083e4f2c1aa32bd (table test-table [id=2b72e7404c48403d86f3ebff68d54dd0]) successfully deleted
W20260619 16:44:04.787334 14865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:04.787631 14865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:04.787748 14865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:04.792019 14865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:04.792098 14865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:04.792162 14865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:44:04.798098 14865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40345
--tserver_master_addrs=127.0.98.62:34767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:44:04.799264 14865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:04.800437 14865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:04.810748 14878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:04.810792 14875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:04.810766 14876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:04.810956 14865 server_base.cc:1034] running on GCE node
I20260619 16:44:04.811508 14865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:04.812212 14865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:04.813392 14865 hybrid_clock.cc:648] HybridClock initialized: now 1781887444813363 us; error 55 us; skew 500 ppm
Jun 19 16:44:04 dist-test-slave-03cd krb5kdc[14092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:04.816351 14865 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:44:04.817322 14865 webserver.cc:460] Webserver started at http://127.0.98.2:40345/ using document root <none> and password file <none>
I20260619 16:44:04.817919 14865 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:04.818029 14865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:04.822100 14865 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:44:04.824407 14885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:04.825788 14865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:44:04.825940 14865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "ae08116dded4430bb72b9c0a76b69d8c"
format_stamp: "Formatted at 2026-06-19 16:44:03 on dist-test-slave-03cd"
server_key: "cf0d6dec595e5e643a67363dffb9c331"
server_key_iv: "3a890b9b23bf8bc931b7c31fe937e7be"
server_key_version: "encryptionkey@0"
I20260619 16:44:04.826527 14865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:04.867614 14865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:04.871238 14865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:04.871466 14865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:04.872131 14865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:44:04.872994 14865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:44:04.873085 14865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:04.873139 14865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:44:04.873207 14865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:04.883370 14865 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43683
I20260619 16:44:04.883540 14998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43683 every 8 connection(s)
I20260619 16:44:04.884635 14865 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:44:04.888011 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 14865
I20260619 16:44:04.888543 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45541
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34767
--webserver_port=45009
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:44:04 dist-test-slave-03cd krb5kdc[14092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:44:04.901661 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.886649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42735 (local address 127.0.98.62:34767)
0619 16:44:04.886849 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:04.886853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:04.887731 (+ 878us) server_negotiation.cc:407] Connection header received
0619 16:44:04.889814 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:04.889817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:04.889888 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:04.889991 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:04.891358 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.892313 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.893680 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.894151 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.894293 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.897322 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:04.897326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:04.897328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:04.897360 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:04.899126 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:04.899842 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:04.899846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:04.899850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:04.899941 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:04.900341 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:04.900344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:04.900346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:04.900559 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:04.900671 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:04.901328 (+ 657us) server_negotiation.cc:299] Negotiation successful
0619 16:44:04.901486 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 16:44:04.902616 15001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.887035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:42735)
0619 16:44:04.887534 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:04.887579 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:04.889479 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:04.890280 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:04.890287 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:04.890692 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:04.891194 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.891208 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.892480 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.892484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.893527 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.893538 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.894414 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.894432 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.894608 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.895336 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:04.895354 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:04.897098 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:04.899284 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:04.899291 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:04.899312 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:04.899721 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:04.900051 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:04.900054 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:04.900057 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:04.900230 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:04.900671 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:04.900680 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:04.900972 (+ 292us) client_negotiation.cc:769] Sending connection context
0619 16:44:04.901261 (+ 289us) client_negotiation.cc:240] Negotiation successful
0619 16:44:04.901671 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":116,"threads_started":1}
I20260619 16:44:04.903959 14999 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34767
I20260619 16:44:04.904227 14999 heartbeater.cc:464] Registering TS with master...
I20260619 16:44:04.904820 14999 heartbeater.cc:511] Master 127.0.98.62:34767 requested a full tablet report, sending...
I20260619 16:44:04.906118 14671 ts_manager.cc:194] Registered new tserver with Master: ae08116dded4430bb72b9c0a76b69d8c (127.0.98.2:43683)
I20260619 16:44:04.906900 14671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42735
W20260619 16:44:05.008809 15002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:05.009100 15002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:05.009151 15002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:05.013336 15002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:05.013415 15002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:05.013478 15002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:44:05.019245 15002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45009
--tserver_master_addrs=127.0.98.62:34767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:44:05.020470 15002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:05.021512 15002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:05.032100 15008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:05.032101 15007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:05.032100 15010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:05.032407 15002 server_base.cc:1034] running on GCE node
I20260619 16:44:05.032907 15002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:05.033607 15002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:05.034812 15002 hybrid_clock.cc:648] HybridClock initialized: now 1781887445034798 us; error 37 us; skew 500 ppm
Jun 19 16:44:05 dist-test-slave-03cd krb5kdc[14092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:05.038154 15002 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:44:05.039109 15002 webserver.cc:460] Webserver started at http://127.0.98.3:45009/ using document root <none> and password file <none>
I20260619 16:44:05.039775 15002 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:05.039865 15002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:05.044263 15002 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:44:05.046627 15017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:05.048156 15002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:44:05.048300 15002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "6791ec79dd4d44b084a2221643274df6"
format_stamp: "Formatted at 2026-06-19 16:44:03 on dist-test-slave-03cd"
server_key: "4a84e24a3cf93c41985099ecf5eb1063"
server_key_iv: "56c2e3f95d6b3cebca369d78f98e9b64"
server_key_version: "encryptionkey@0"
I20260619 16:44:05.048879 15002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:05.074020 15002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:05.077412 15002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:05.077631 15002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:05.078320 15002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:44:05.079177 15002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:44:05.079231 15002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:05.079295 15002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:44:05.079329 15002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:05.088490 15002 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45541
I20260619 16:44:05.088510 15130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45541 every 8 connection(s)
I20260619 16:44:05.089507 15002 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:44:05.096642 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 15002
Jun 19 16:44:05 dist-test-slave-03cd krb5kdc[14092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:44:05.105274 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:05.091397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52103 (local address 127.0.98.62:34767)
0619 16:44:05.091569 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:05.091574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:05.092476 (+ 902us) server_negotiation.cc:407] Connection header received
0619 16:44:05.093786 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:05.093789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:05.093854 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:05.093936 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:05.095265 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.095986 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.097702 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.098123 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.098227 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:05.101428 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:05.101431 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:05.101433 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:05.101450 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:05.103074 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:05.103591 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:05.103594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:05.103595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:05.103730 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:05.104120 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:05.104125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:05.104127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:05.104310 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:05.104418 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:05.104966 (+ 548us) server_negotiation.cc:299] Negotiation successful
0619 16:44:05.105100 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 16:44:05.106458 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:05.097411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34750 (local address 127.0.98.62:34767)
0619 16:44:05.097792 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:05.097797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:05.097809 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:44:05.097968 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:05.097971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:05.098023 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:05.098118 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:05.099280 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.099959 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.100915 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.101145 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.101224 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:05.102693 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:05.102697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:05.102699 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:05.102742 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:05.104643 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:05.105083 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:05.105088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:05.105090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:05.105172 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:05.105479 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:05.105485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:05.105488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:05.105689 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:05.105849 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:05.106120 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 16:44:05.106239 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":79,"threads_started":1}
I20260619 16:44:05.106469 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:05.091860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:52103)
0619 16:44:05.092316 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:05.092358 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:05.093552 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:05.094117 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:05.094123 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:05.094553 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:05.095090 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:05.095105 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.096138 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.096141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:05.097536 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:05.097549 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.098308 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.098314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:05.098574 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:05.099488 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:05.099506 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:05.101259 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:05.103186 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:05.103191 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:05.103206 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:05.103490 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:05.103841 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:05.103844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:05.103846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:05.104010 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:05.104421 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:05.104430 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:05.104719 (+ 289us) client_negotiation.cc:769] Sending connection context
0619 16:44:05.104926 (+ 207us) client_negotiation.cc:240] Negotiation successful
0619 16:44:05.105301 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":121,"threads_started":1}
I20260619 16:44:05.107789 15131 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34767
I20260619 16:44:05.108059 15131 heartbeater.cc:464] Registering TS with master...
I20260619 16:44:05.108640 15131 heartbeater.cc:511] Master 127.0.98.62:34767 requested a full tablet report, sending...
I20260619 16:44:05.109715 14671 ts_manager.cc:194] Registered new tserver with Master: 6791ec79dd4d44b084a2221643274df6 (127.0.98.3:45541)
I20260619 16:44:05.110294 14671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52103
I20260619 16:44:05.120707 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:44:05.125293 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:05.121360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34754 (local address 127.0.98.62:34767)
0619 16:44:05.121523 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:05.121529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:05.121685 (+ 156us) server_negotiation.cc:407] Connection header received
0619 16:44:05.121828 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:05.121830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:05.121941 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:05.122032 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:44:05.122261 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.122799 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.123626 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.123941 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.124032 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:05.124362 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:44:05.124820 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:44:05.124924 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:05.125015 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 16:44:05.125074 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:44:05.126068 14671 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:34754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:44:05.129606 14671 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:44:05.138845 14795 tablet_service.cc:1467] Processing CreateTablet for tablet 2e0f698e48d847829bb2e669f887aa81 (DEFAULT_TABLE table=test-table [id=e86f7d99f7db416d91ad746c6d098057]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:44:05.139534 14795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e0f698e48d847829bb2e669f887aa81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:05.142679 14753 tablet_bootstrap.cc:492] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942: Bootstrap starting.
I20260619 16:44:05.142747 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:05.137369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45541 (local address 127.0.0.1:60292)
0619 16:44:05.137506 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:05.137524 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:05.137612 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:05.138501 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:05.138504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:05.138525 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:05.138639 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:05.138646 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.140074 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.140081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:05.141096 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:05.141107 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.142220 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.142223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:05.142323 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:05.142387 (+ 64us) client_negotiation.cc:769] Sending connection context
0619 16:44:05.142475 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 16:44:05.142593 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":60}
I20260619 16:44:05.142848 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:05.137720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43683 (local address 127.0.0.1:54322)
0619 16:44:05.137995 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:05.138010 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:05.138098 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:05.138816 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:05.138819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:05.138835 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:05.138932 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:05.138938 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.140133 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.140139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:05.141330 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:05.141342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.142379 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.142385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:05.142519 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:05.142546 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 16:44:05.142599 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 16:44:05.142672 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":83,"threads_started":1}
I20260619 16:44:05.143527 15142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:05.137925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54322 (local address 127.0.98.2:43683)
0619 16:44:05.138358 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:05.138371 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:05.138395 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:44:05.138475 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:05.138489 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:05.138661 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:05.138837 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:05.139093 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.139987 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.141466 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.142239 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.143038 (+ 799us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:05.143190 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:05.143238 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 16:44:05.143350 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":106,"threads_started":1}
I20260619 16:44:05.143703 14753 tablet_bootstrap.cc:654] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:05.143811 15140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:05.137553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60292 (local address 127.0.98.3:45541)
0619 16:44:05.137930 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:05.137943 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:05.137970 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:44:05.138060 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:05.138066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:05.138323 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:05.138502 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:05.138770 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.139910 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.141236 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.142059 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.142182 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:05.142358 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:05.142530 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 16:44:05.143634 (+ 1104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":129,"threads_started":1}
I20260619 16:44:05.145294 14753 tablet_bootstrap.cc:492] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942: No bootstrap required, opened a new log
I20260619 16:44:05.145395 14753 ts_tablet_manager.cc:1397] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 16:44:05.145880 14753 raft_consensus.cc:348] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6791ec79dd4d44b084a2221643274df6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45541 } } peers { permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 } } peers { permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 } }
I20260619 16:44:05.145999 14753 raft_consensus.cc:374] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:05.146040 14753 raft_consensus.cc:729] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2e3597861584b1ba48bd52fe0489942, State: Initialized, Role: FOLLOWER
I20260619 16:44:05.146378 14753 consensus_queue.cc:260] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6791ec79dd4d44b084a2221643274df6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45541 } } peers { permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 } } peers { permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 } }
I20260619 16:44:05.146771 14753 ts_tablet_manager.cc:1428] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 16:44:05.147084 14933 tablet_service.cc:1467] Processing CreateTablet for tablet 2e0f698e48d847829bb2e669f887aa81 (DEFAULT_TABLE table=test-table [id=e86f7d99f7db416d91ad746c6d098057]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:44:05.147082 15065 tablet_service.cc:1467] Processing CreateTablet for tablet 2e0f698e48d847829bb2e669f887aa81 (DEFAULT_TABLE table=test-table [id=e86f7d99f7db416d91ad746c6d098057]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:44:05.148231 14933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e0f698e48d847829bb2e669f887aa81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:05.148288 15065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e0f698e48d847829bb2e669f887aa81. 1 dirs total, 0 dirs full, 0 dirs failed
W20260619 16:44:05.153438 14862 tablet.cc:2369] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:44:05.155840 15145 tablet_bootstrap.cc:492] T 2e0f698e48d847829bb2e669f887aa81 P ae08116dded4430bb72b9c0a76b69d8c: Bootstrap starting.
I20260619 16:44:05.158156 15145 tablet_bootstrap.cc:654] T 2e0f698e48d847829bb2e669f887aa81 P ae08116dded4430bb72b9c0a76b69d8c: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:05.158627 15144 tablet_bootstrap.cc:492] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6: Bootstrap starting.
I20260619 16:44:05.159027 15145 log.cc:824] T 2e0f698e48d847829bb2e669f887aa81 P ae08116dded4430bb72b9c0a76b69d8c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:44:05.160717 15144 tablet_bootstrap.cc:654] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:05.161139 15145 tablet_bootstrap.cc:492] T 2e0f698e48d847829bb2e669f887aa81 P ae08116dded4430bb72b9c0a76b69d8c: No bootstrap required, opened a new log
I20260619 16:44:05.161358 15145 ts_tablet_manager.cc:1397] T 2e0f698e48d847829bb2e669f887aa81 P ae08116dded4430bb72b9c0a76b69d8c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 16:44:05.161835 15144 log.cc:824] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6: Log is configured to *not* fsync() on all Append() calls
I20260619 16:44:05.163923 15144 tablet_bootstrap.cc:492] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6: No bootstrap required, opened a new log
I20260619 16:44:05.164146 15144 ts_tablet_manager.cc:1397] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 16:44:05.165004 15145 raft_consensus.cc:348] T 2e0f698e48d847829bb2e669f887aa81 P ae08116dded4430bb72b9c0a76b69d8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6791ec79dd4d44b084a2221643274df6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45541 } } peers { permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 } } peers { permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 } }
I20260619 16:44:05.165309 15145 raft_consensus.cc:374] T 2e0f698e48d847829bb2e669f887aa81 P ae08116dded4430bb72b9c0a76b69d8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:05.165372 15145 raft_consensus.cc:729] T 2e0f698e48d847829bb2e669f887aa81 P ae08116dded4430bb72b9c0a76b69d8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae08116dded4430bb72b9c0a76b69d8c, State: Initialized, Role: FOLLOWER
I20260619 16:44:05.165838 15145 consensus_queue.cc:260] T 2e0f698e48d847829bb2e669f887aa81 P ae08116dded4430bb72b9c0a76b69d8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6791ec79dd4d44b084a2221643274df6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45541 } } peers { permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 } } peers { permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 } }
I20260619 16:44:05.166550 14999 heartbeater.cc:503] Master 127.0.98.62:34767 was elected leader, sending a full tablet report...
I20260619 16:44:05.166875 15145 ts_tablet_manager.cc:1428] T 2e0f698e48d847829bb2e669f887aa81 P ae08116dded4430bb72b9c0a76b69d8c: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260619 16:44:05.167845 15144 raft_consensus.cc:348] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6791ec79dd4d44b084a2221643274df6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45541 } } peers { permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 } } peers { permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 } }
I20260619 16:44:05.168095 15144 raft_consensus.cc:374] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:05.168133 15144 raft_consensus.cc:729] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6791ec79dd4d44b084a2221643274df6, State: Initialized, Role: FOLLOWER
I20260619 16:44:05.168576 15144 consensus_queue.cc:260] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6791ec79dd4d44b084a2221643274df6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45541 } } peers { permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 } } peers { permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 } }
I20260619 16:44:05.169317 15131 heartbeater.cc:503] Master 127.0.98.62:34767 was elected leader, sending a full tablet report...
I20260619 16:44:05.169628 15144 ts_tablet_manager.cc:1428] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 16:44:05.307176 15149 raft_consensus.cc:482] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:44:05.307438 15149 raft_consensus.cc:504] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6791ec79dd4d44b084a2221643274df6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45541 } } peers { permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 } } peers { permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 } }
I20260619 16:44:05.308763 15149 leader_election.cc:288] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae08116dded4430bb72b9c0a76b69d8c (127.0.98.2:43683), c2e3597861584b1ba48bd52fe0489942 (127.0.98.1:45375)
I20260619 16:44:05.313900 15142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:05.309078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37269 (local address 127.0.98.2:43683)
0619 16:44:05.309207 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:05.309214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:05.309233 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:44:05.309458 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:05.309465 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:05.309549 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:05.309645 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:05.309932 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.310708 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.312490 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.313098 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.313225 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:05.313302 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:05.313689 (+ 387us) server_negotiation.cc:299] Negotiation successful
0619 16:44:05.313763 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:44:05.313902 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:05.309167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37995 (local address 127.0.98.1:45375)
0619 16:44:05.309450 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:05.309455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:05.309504 (+ 49us) server_negotiation.cc:407] Connection header received
0619 16:44:05.309624 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:05.309630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:05.309695 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:05.309810 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:05.310132 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.310859 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.312482 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.313167 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.313261 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:05.313423 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:05.313689 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 16:44:05.313763 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":92,"threads_started":1}
I20260619 16:44:05.313915 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:05.308988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43683 (local address 127.0.98.3:37269)
0619 16:44:05.309177 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:05.309193 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:05.309295 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:05.309666 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:05.309668 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:05.309681 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:05.309780 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:05.309809 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.310832 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.310835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:05.312341 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:05.312359 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.313262 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.313266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:05.313539 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:05.313578 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 16:44:05.313649 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 16:44:05.313747 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":189}
I20260619 16:44:05.313971 15150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:05.309097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45375 (local address 127.0.98.3:37995)
0619 16:44:05.309436 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:05.309447 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:05.309519 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:05.309846 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:05.309901 (+ 55us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:05.309929 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:05.310014 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:05.310019 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.310963 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.310966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:05.312346 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:05.312353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.313274 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.313277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:05.313520 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:05.313565 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 16:44:05.313649 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 16:44:05.313747 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":177,"thread_start_us":138,"threads_started":1}
I20260619 16:44:05.314709 14953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0f698e48d847829bb2e669f887aa81" candidate_uuid: "6791ec79dd4d44b084a2221643274df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae08116dded4430bb72b9c0a76b69d8c" is_pre_election: true
I20260619 16:44:05.315078 14953 raft_consensus.cc:2454] T 2e0f698e48d847829bb2e669f887aa81 P ae08116dded4430bb72b9c0a76b69d8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6791ec79dd4d44b084a2221643274df6 in term 0.
I20260619 16:44:05.315107 14815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0f698e48d847829bb2e669f887aa81" candidate_uuid: "6791ec79dd4d44b084a2221643274df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2e3597861584b1ba48bd52fe0489942" is_pre_election: true
I20260619 16:44:05.315449 14815 raft_consensus.cc:2454] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6791ec79dd4d44b084a2221643274df6 in term 0.
I20260619 16:44:05.315631 15019 leader_election.cc:302] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6791ec79dd4d44b084a2221643274df6, ae08116dded4430bb72b9c0a76b69d8c; no voters:
I20260619 16:44:05.316006 15149 raft_consensus.cc:2781] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:44:05.316090 15149 raft_consensus.cc:482] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:44:05.316123 15149 raft_consensus.cc:3037] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:05.317220 15149 raft_consensus.cc:504] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6791ec79dd4d44b084a2221643274df6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45541 } } peers { permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 } } peers { permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 } }
I20260619 16:44:05.317619 15149 leader_election.cc:288] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6 [CANDIDATE]: Term 1 election: Requested vote from peers ae08116dded4430bb72b9c0a76b69d8c (127.0.98.2:43683), c2e3597861584b1ba48bd52fe0489942 (127.0.98.1:45375)
I20260619 16:44:05.318089 14953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0f698e48d847829bb2e669f887aa81" candidate_uuid: "6791ec79dd4d44b084a2221643274df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae08116dded4430bb72b9c0a76b69d8c"
I20260619 16:44:05.318238 14953 raft_consensus.cc:3037] T 2e0f698e48d847829bb2e669f887aa81 P ae08116dded4430bb72b9c0a76b69d8c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:05.318326 14815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0f698e48d847829bb2e669f887aa81" candidate_uuid: "6791ec79dd4d44b084a2221643274df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2e3597861584b1ba48bd52fe0489942"
I20260619 16:44:05.318500 14815 raft_consensus.cc:3037] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:05.319413 14953 raft_consensus.cc:2454] T 2e0f698e48d847829bb2e669f887aa81 P ae08116dded4430bb72b9c0a76b69d8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6791ec79dd4d44b084a2221643274df6 in term 1.
I20260619 16:44:05.319648 14815 raft_consensus.cc:2454] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6791ec79dd4d44b084a2221643274df6 in term 1.
I20260619 16:44:05.319783 15019 leader_election.cc:302] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6791ec79dd4d44b084a2221643274df6, ae08116dded4430bb72b9c0a76b69d8c; no voters:
I20260619 16:44:05.319989 15149 raft_consensus.cc:2781] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:44:05.320256 15149 raft_consensus.cc:686] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6 [term 1 LEADER]: Becoming Leader. State: Replica: 6791ec79dd4d44b084a2221643274df6, State: Running, Role: LEADER
I20260619 16:44:05.320642 15149 consensus_queue.cc:237] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6791ec79dd4d44b084a2221643274df6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45541 } } peers { permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 } } peers { permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 } }
I20260619 16:44:05.324044 14671 catalog_manager.cc:5495] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6791ec79dd4d44b084a2221643274df6 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6791ec79dd4d44b084a2221643274df6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6791ec79dd4d44b084a2221643274df6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 } health_report { overall_health: UNKNOWN } } }
W20260619 16:44:05.340512 15132 tablet.cc:2369] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:44:05.340523 15140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:05.336713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60294 (local address 127.0.98.3:45541)
0619 16:44:05.336874 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:05.336879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:05.336895 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:44:05.337032 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:05.337035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:05.337101 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:05.337223 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:44:05.337460 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.337982 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.338850 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.339062 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.339176 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:05.339492 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:44:05.340075 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:44:05.340208 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:05.340270 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 16:44:05.340336 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:44:05.347046 14815 raft_consensus.cc:1261] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [term 1 FOLLOWER]: Refusing update from remote peer 6791ec79dd4d44b084a2221643274df6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:44:05.347056 14953 raft_consensus.cc:1261] T 2e0f698e48d847829bb2e669f887aa81 P ae08116dded4430bb72b9c0a76b69d8c [term 1 FOLLOWER]: Refusing update from remote peer 6791ec79dd4d44b084a2221643274df6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:44:05.347918 15152 consensus_queue.cc:1035] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:44:05.348112 15149 consensus_queue.cc:1035] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:44:05.364319 14672 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:34754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:44:05.364543 14672 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:34754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:44:05.367260 14672 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331: Sending DeleteTablet for 3 replicas of tablet 2e0f698e48d847829bb2e669f887aa81
I20260619 16:44:05.368196 14933 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e0f698e48d847829bb2e669f887aa81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54322
I20260619 16:44:05.368191 15065 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e0f698e48d847829bb2e669f887aa81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60292
I20260619 16:44:05.368798 15163 tablet_replica.cc:330] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6: stopping tablet replica
I20260619 16:44:05.368772 14795 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e0f698e48d847829bb2e669f887aa81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38378
I20260619 16:44:05.368986 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 14711
I20260619 16:44:05.369148 15165 tablet_replica.cc:330] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942: stopping tablet replica
I20260619 16:44:05.369155 15164 tablet_replica.cc:330] T 2e0f698e48d847829bb2e669f887aa81 P ae08116dded4430bb72b9c0a76b69d8c: stopping tablet replica
I20260619 16:44:05.369294 15165 raft_consensus.cc:2229] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:44:05.369465 15165 raft_consensus.cc:2258] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:44:05.369835 15163 raft_consensus.cc:2229] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:44:05.370096 15164 raft_consensus.cc:2229] T 2e0f698e48d847829bb2e669f887aa81 P ae08116dded4430bb72b9c0a76b69d8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:44:05.370489 15164 raft_consensus.cc:2258] T 2e0f698e48d847829bb2e669f887aa81 P ae08116dded4430bb72b9c0a76b69d8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:44:05.370813 15163 raft_consensus.cc:2258] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:44:05.372731 15164 ts_tablet_manager.cc:1905] T 2e0f698e48d847829bb2e669f887aa81 P ae08116dded4430bb72b9c0a76b69d8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:44:05.372731 15163 ts_tablet_manager.cc:1905] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:44:05.375896 15163 ts_tablet_manager.cc:1918] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:44:05.375896 15164 ts_tablet_manager.cc:1918] T 2e0f698e48d847829bb2e669f887aa81 P ae08116dded4430bb72b9c0a76b69d8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:44:05.376031 15163 log.cc:1196] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/wal/wals/2e0f698e48d847829bb2e669f887aa81
I20260619 16:44:05.376183 15164 log.cc:1196] T 2e0f698e48d847829bb2e669f887aa81 P ae08116dded4430bb72b9c0a76b69d8c: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/wal/wals/2e0f698e48d847829bb2e669f887aa81
I20260619 16:44:05.376410 15163 ts_tablet_manager.cc:1939] T 2e0f698e48d847829bb2e669f887aa81 P 6791ec79dd4d44b084a2221643274df6: Deleting consensus metadata
I20260619 16:44:05.376644 15164 ts_tablet_manager.cc:1939] T 2e0f698e48d847829bb2e669f887aa81 P ae08116dded4430bb72b9c0a76b69d8c: Deleting consensus metadata
I20260619 16:44:05.377840 14658 catalog_manager.cc:4841] TS ae08116dded4430bb72b9c0a76b69d8c (127.0.98.2:43683): tablet 2e0f698e48d847829bb2e669f887aa81 (table test-table [id=e86f7d99f7db416d91ad746c6d098057]) successfully deleted
I20260619 16:44:05.378136 14659 catalog_manager.cc:4841] TS 6791ec79dd4d44b084a2221643274df6 (127.0.98.3:45541): tablet 2e0f698e48d847829bb2e669f887aa81 (table test-table [id=e86f7d99f7db416d91ad746c6d098057]) successfully deleted
W20260619 16:44:05.378374 14659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45375 (error 108)
I20260619 16:44:05.378861 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 14865
I20260619 16:44:05.380288 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:05.379927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45375 (local address 127.0.0.1:38392)
0619 16:44:05.380085 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:05.380173 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 16:44:05.380486 14659 catalog_manager.cc:4584] TS c2e3597861584b1ba48bd52fe0489942 (127.0.98.1:45375): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2e0f698e48d847829bb2e669f887aa81: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45375: connect: Connection refused (error 111)
I20260619 16:44:05.385264 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 15002
I20260619 16:44:05.390995 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 14640
I20260619 16:44:05.397431 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34767
--webserver_interface=127.0.98.62
--webserver_port=37499
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:44:05.511053 15167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:05.511364 15167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:05.511415 15167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:05.515632 15167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:44:05.515733 15167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:05.515754 15167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:05.515771 15167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:44:05.515787 15167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:44:05.521416 15167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:44:05.522540 15167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:05.523622 15167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:05.532481 15172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:05.532475 15173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:05.532575 15175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:05.532770 15167 server_base.cc:1034] running on GCE node
I20260619 16:44:05.533492 15167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:05.534703 15167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:05.535892 15167 hybrid_clock.cc:648] HybridClock initialized: now 1781887445535862 us; error 51 us; skew 500 ppm
Jun 19 16:44:05 dist-test-slave-03cd krb5kdc[14092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:05.538790 15167 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:44:05.539826 15167 webserver.cc:460] Webserver started at http://127.0.98.62:37499/ using document root <none> and password file <none>
I20260619 16:44:05.540421 15167 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:05.540529 15167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:05.544786 15167 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260619 16:44:05.547197 15182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:05.548822 15167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:44:05.549011 15167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "e06d30f6f6d24164b9db02ecfed70331"
format_stamp: "Formatted at 2026-06-19 16:44:03 on dist-test-slave-03cd"
server_key: "baa2404c716818f086beba88a69cfdfa"
server_key_iv: "7d884047083ab07b9f332bf4a7b5519f"
server_key_version: "encryptionkey@0"
I20260619 16:44:05.549656 15167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:05.566792 15167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:05.570616 15167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:05.570858 15167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:05.578351 15167 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34767
I20260619 16:44:05.578673 15234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34767 every 8 connection(s)
I20260619 16:44:05.579571 15167 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:44:05.584329 15235 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:44:05.584713 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 15167
I20260619 16:44:05.585189 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45375
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34767
--webserver_port=36737
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:44:05.586853 15235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331: Bootstrap starting.
I20260619 16:44:05.605103 15235 log.cc:824] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331: Log is configured to *not* fsync() on all Append() calls
I20260619 16:44:05.616719 15235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:44:05.617172 15235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331: Bootstrap complete.
I20260619 16:44:05.621438 15235 raft_consensus.cc:348] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06d30f6f6d24164b9db02ecfed70331" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } }
I20260619 16:44:05.621872 15235 raft_consensus.cc:729] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e06d30f6f6d24164b9db02ecfed70331, State: Initialized, Role: FOLLOWER
I20260619 16:44:05.622493 15235 consensus_queue.cc:260] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06d30f6f6d24164b9db02ecfed70331" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } }
I20260619 16:44:05.622620 15235 raft_consensus.cc:388] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:44:05.622720 15235 raft_consensus.cc:482] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:44:05.622848 15235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [term 2 FOLLOWER]: Advancing to term 3
I20260619 16:44:05.624969 15235 raft_consensus.cc:504] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06d30f6f6d24164b9db02ecfed70331" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } }
I20260619 16:44:05.625339 15235 leader_election.cc:302] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e06d30f6f6d24164b9db02ecfed70331; no voters:
I20260619 16:44:05.625697 15235 leader_election.cc:288] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 16:44:05.625852 15239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 16:44:05.626108 15239 raft_consensus.cc:686] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [term 3 LEADER]: Becoming Leader. State: Replica: e06d30f6f6d24164b9db02ecfed70331, State: Running, Role: LEADER
I20260619 16:44:05.626433 15239 consensus_queue.cc:237] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06d30f6f6d24164b9db02ecfed70331" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } }
I20260619 16:44:05.626981 15235 sys_catalog.cc:564] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:44:05.628003 15240 sys_catalog.cc:455] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e06d30f6f6d24164b9db02ecfed70331" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06d30f6f6d24164b9db02ecfed70331" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } } }
I20260619 16:44:05.628037 15241 sys_catalog.cc:455] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e06d30f6f6d24164b9db02ecfed70331. Latest consensus state: current_term: 3 leader_uuid: "e06d30f6f6d24164b9db02ecfed70331" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06d30f6f6d24164b9db02ecfed70331" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } } }
I20260619 16:44:05.628165 15240 sys_catalog.cc:458] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [sys.catalog]: This master's current role is: LEADER
I20260619 16:44:05.628166 15241 sys_catalog.cc:458] T 00000000000000000000000000000000 P e06d30f6f6d24164b9db02ecfed70331 [sys.catalog]: This master's current role is: LEADER
I20260619 16:44:05.628706 15248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:44:05.634320 15248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:44:05.635193 15248 catalog_manager.cc:1259] Loaded cluster ID: 57073abe06e945c59849491335a9afb0
I20260619 16:44:05.635252 15248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 16:44:05.636293 15248 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5a7c8dd9ad google::LogMessage::Fail() at ??:0
@ 0x7f5a7c8e088a google::LogMessage::SendToLog() at ??:0
@ 0x7f5a7c8dd4de google::LogMessage::Flush() at ??:0
@ 0x7f5a7c8dea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5a8721cfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f5a8721dd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f5a8721a579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f5a8725e640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f5a7eb2f3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5a7eb2fc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f5a7eb313f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f5a7eb20512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5a801646db start_thread at ??:0
@ 0x7f5a7b3db61f clone at ??:0
W20260619 16:44:05.737964 15237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:05.738373 15237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:05.738440 15237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:05.744928 15237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:05.745035 15237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:05.745127 15237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:44:05.753433 15237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36737
--tserver_master_addrs=127.0.98.62:34767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:44:05.754948 15237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:05.756335 15237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:05.768265 15277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:05.768427 15237 server_base.cc:1034] running on GCE node
W20260619 16:44:05.768275 15279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:05.768275 15276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:05.769104 15237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:05.769802 15237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:05.770996 15237 hybrid_clock.cc:648] HybridClock initialized: now 1781887445770970 us; error 47 us; skew 500 ppm
Jun 19 16:44:05 dist-test-slave-03cd krb5kdc[14092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:05.774111 15237 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:44:05.775144 15237 webserver.cc:460] Webserver started at http://127.0.98.1:36737/ using document root <none> and password file <none>
I20260619 16:44:05.775795 15237 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:05.775864 15237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:05.780254 15237 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:44:05.783036 15286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:05.784801 15237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:44:05.784919 15237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "c2e3597861584b1ba48bd52fe0489942"
format_stamp: "Formatted at 2026-06-19 16:44:03 on dist-test-slave-03cd"
server_key: "4bdc678131716693d680eb44e7b4744f"
server_key_iv: "3d90c0b223a987bebbadedf5e7ced098"
server_key_version: "encryptionkey@0"
I20260619 16:44:05.785642 15237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:05.823945 15237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:05.827545 15237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:05.827836 15237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:05.828471 15237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:44:05.829656 15293 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 16:44:05.834067 15237 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 16:44:05.834192 15237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260619 16:44:05.834292 15237 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 16:44:05.837438 15237 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 16:44:05.837522 15237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.001s
I20260619 16:44:05.837896 15293 tablet_bootstrap.cc:492] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942: Bootstrap starting.
I20260619 16:44:05.849711 15237 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45375
I20260619 16:44:05.849869 15400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45375 every 8 connection(s)
I20260619 16:44:05.850987 15237 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:44:05.854044 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 15237
I20260619 16:44:05.854524 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43683
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34767
--webserver_port=40345
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:44:05.858743 15293 log.cc:824] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942: Log is configured to *not* fsync() on all Append() calls
I20260619 16:44:05.865944 15293 tablet_bootstrap.cc:492] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:44:05.866397 15293 tablet_bootstrap.cc:492] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942: Bootstrap complete.
I20260619 16:44:05.867539 15293 ts_tablet_manager.cc:1397] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942: Time spent bootstrapping tablet: real 0.030s user 0.017s sys 0.009s
I20260619 16:44:05.870107 15293 raft_consensus.cc:348] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6791ec79dd4d44b084a2221643274df6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45541 } } peers { permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 } } peers { permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 } }
I20260619 16:44:05.870599 15293 raft_consensus.cc:729] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2e3597861584b1ba48bd52fe0489942, State: Initialized, Role: FOLLOWER
I20260619 16:44:05.871140 15293 consensus_queue.cc:260] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6791ec79dd4d44b084a2221643274df6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45541 } } peers { permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 } } peers { permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 } }
I20260619 16:44:05.872192 15293 ts_tablet_manager.cc:1428] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260619 16:44:05.919332 15403 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:05.853469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:42359)
0619 16:44:05.854918 (+ 1449us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:05.854997 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:05.856128 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:05.918441 (+ 62313us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1062,"thread_start_us":161,"threads_started":1}
W20260619 16:44:05.920063 15401 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34767 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34767: Client connection negotiation failed: client connection to 127.0.98.62:34767: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 16:44:05.920866 15403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:05.920575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:53313)
0619 16:44:05.920714 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:05.920798 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 16:44:05.921617 15403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:05.921388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:56391)
0619 16:44:05.921508 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:05.921555 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 16:44:05.921887 15401 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:44:05.991776 15404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:05.992159 15404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:05.992269 15404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:05.996627 15404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:05.996706 15404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:05.996771 15404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:44:06.002406 15404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40345
--tserver_master_addrs=127.0.98.62:34767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:44:06.003644 15404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:06.004789 15404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:06.015893 15411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:06.015893 15412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:06.015895 15414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:06.015941 15404 server_base.cc:1034] running on GCE node
I20260619 16:44:06.016666 15404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:06.017364 15404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:06.018589 15404 hybrid_clock.cc:648] HybridClock initialized: now 1781887446018541 us; error 50 us; skew 500 ppm
Jun 19 16:44:06 dist-test-slave-03cd krb5kdc[14092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:06.021852 15404 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:44:06.022899 15404 webserver.cc:460] Webserver started at http://127.0.98.2:40345/ using document root <none> and password file <none>
I20260619 16:44:06.023530 15404 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:06.023631 15404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:06.027833 15404 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 16:44:06.030134 15421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:06.031530 15404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:44:06.031720 15404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "ae08116dded4430bb72b9c0a76b69d8c"
format_stamp: "Formatted at 2026-06-19 16:44:03 on dist-test-slave-03cd"
server_key: "cf0d6dec595e5e643a67363dffb9c331"
server_key_iv: "3a890b9b23bf8bc931b7c31fe937e7be"
server_key_version: "encryptionkey@0"
I20260619 16:44:06.032316 15404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:06.047575 15404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:06.050765 15404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:06.050935 15404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:06.051491 15404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:44:06.052371 15404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:44:06.052421 15404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:06.052462 15404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:44:06.052479 15404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:06.062181 15404 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43683
I20260619 16:44:06.062245 15534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43683 every 8 connection(s)
I20260619 16:44:06.063308 15404 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:44:06.064602 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 15404
I20260619 16:44:06.065133 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45541
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34767
--webserver_port=45009
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:44:06.067984 15538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:06.065997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:42855)
0619 16:44:06.066504 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:06.066641 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":151,"threads_started":1}
W20260619 16:44:06.068624 15535 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34767 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34767: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
I20260619 16:44:06.069407 15538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:06.069144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:47689)
0619 16:44:06.069282 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:06.069340 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 16:44:06.070114 15538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:06.069869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:47517)
0619 16:44:06.070015 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:06.070060 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 16:44:06.070282 15535 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:44:06.180042 15537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:06.180404 15537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:06.180490 15537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:06.184814 15537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:06.184942 15537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:06.185050 15537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:44:06.190927 15537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45009
--tserver_master_addrs=127.0.98.62:34767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:44:06.192229 15537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:06.193295 15537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:44:06.203799 15537 server_base.cc:1034] running on GCE node
W20260619 16:44:06.203723 15543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:06.203704 15544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:06.203660 15546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:06.204516 15537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:06.205229 15537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:06.206410 15537 hybrid_clock.cc:648] HybridClock initialized: now 1781887446206392 us; error 35 us; skew 500 ppm
Jun 19 16:44:06 dist-test-slave-03cd krb5kdc[14092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:06.209488 15537 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:44:06.210501 15537 webserver.cc:460] Webserver started at http://127.0.98.3:45009/ using document root <none> and password file <none>
I20260619 16:44:06.211059 15537 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:06.211118 15537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:06.215226 15537 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:44:06.217730 15553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:06.219161 15537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:44:06.219338 15537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "6791ec79dd4d44b084a2221643274df6"
format_stamp: "Formatted at 2026-06-19 16:44:03 on dist-test-slave-03cd"
server_key: "4a84e24a3cf93c41985099ecf5eb1063"
server_key_iv: "56c2e3f95d6b3cebca369d78f98e9b64"
server_key_version: "encryptionkey@0"
I20260619 16:44:06.220082 15537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:06.241946 15537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:06.245559 15537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:06.245813 15537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:06.246524 15537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:44:06.247423 15537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:44:06.247498 15537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:06.247572 15537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:44:06.247615 15537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:06.257752 15537 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45541
I20260619 16:44:06.257781 15666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45541 every 8 connection(s)
I20260619 16:44:06.258795 15537 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:44:06.262856 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 15537
I20260619 16:44:06.263155 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:06.261143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:42311)
0619 16:44:06.261642 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:06.261751 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":122,"threads_started":1}
W20260619 16:44:06.263826 15667 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34767 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34767: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
I20260619 16:44:06.264520 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:06.264287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:57175)
0619 16:44:06.264407 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:06.264455 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 16:44:06.265157 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:06.264966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:60369)
0619 16:44:06.265068 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:06.265109 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 16:44:06.265355 15667 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 16:44:06.923305 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:06.922642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:59041)
0619 16:44:06.923073 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:06.923164 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":121,"threads_started":1}
I20260619 16:44:07.071527 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:07.070886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:41081)
0619 16:44:07.071273 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:07.071374 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":91,"threads_started":1}
I20260619 16:44:07.267990 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:07.267233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:35831)
0619 16:44:07.267726 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:07.267821 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":115,"threads_started":1}
I20260619 16:44:07.430812 15674 raft_consensus.cc:482] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:44:07.431057 15674 raft_consensus.cc:504] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6791ec79dd4d44b084a2221643274df6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45541 } } peers { permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 } } peers { permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 } }
I20260619 16:44:07.432433 15674 leader_election.cc:288] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6791ec79dd4d44b084a2221643274df6 (127.0.98.3:45541), ae08116dded4430bb72b9c0a76b69d8c (127.0.98.2:43683)
Jun 19 16:44:07 dist-test-slave-03cd krb5kdc[14092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 19 16:44:07 dist-test-slave-03cd krb5kdc[14092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260619 16:44:07.452833 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:07.432998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59897 (local address 127.0.98.2:43683)
0619 16:44:07.433515 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:07.433547 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:07.433600 (+ 53us) server_negotiation.cc:407] Connection header received
0619 16:44:07.434661 (+ 1061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:07.434671 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:07.435184 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:07.435595 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:07.438261 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:07.440038 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:07.442352 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:07.443030 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:07.443565 (+ 535us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:07.446854 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:07.446861 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:07.446874 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:07.446911 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:07.449674 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:07.450432 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:07.450442 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:07.450450 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:07.450633 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:07.451085 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:07.451090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:07.451092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:07.451568 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:07.451759 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:07.452351 (+ 592us) server_negotiation.cc:299] Negotiation successful
0619 16:44:07.452624 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":93,"threads_started":1}
I20260619 16:44:07.452816 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:07.432832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43683 (local address 127.0.98.1:59897)
0619 16:44:07.433317 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:07.433340 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:07.433484 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:07.435831 (+ 2347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:07.435840 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:07.436798 (+ 958us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:07.437934 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:07.437966 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:07.440342 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:07.440350 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:07.442141 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:07.442163 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:07.443213 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:07.443217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:07.443398 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:07.444403 (+ 1005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:07.444434 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:07.446578 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:07.449885 (+ 3307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:07.449892 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:07.449911 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:07.450285 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:07.450770 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:07.450774 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:07.450776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:07.450974 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:07.451720 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:07.451727 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:07.452019 (+ 292us) client_negotiation.cc:769] Sending connection context
0619 16:44:07.452226 (+ 207us) client_negotiation.cc:240] Negotiation successful
0619 16:44:07.452604 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":211,"spinlock_wait_cycles":5248,"thread_start_us":110,"threads_started":1}
I20260619 16:44:07.453450 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:07.433025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45541 (local address 127.0.98.1:60685)
0619 16:44:07.433700 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:07.433728 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:07.433886 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:07.435831 (+ 1945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:07.435840 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:07.436798 (+ 958us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:07.437933 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:07.437966 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:07.440350 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:07.440354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:07.441728 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:07.441744 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:07.443049 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:07.443058 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:07.443423 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:07.444554 (+ 1131us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:07.444578 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:07.446926 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:07.449878 (+ 2952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:07.449891 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:07.449917 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:07.450355 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:07.450769 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:07.450772 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:07.450774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:07.451030 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:07.452283 (+ 1253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:07.452286 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:07.452417 (+ 131us) client_negotiation.cc:769] Sending connection context
0619 16:44:07.452544 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 16:44:07.452782 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":468,"spinlock_wait_cycles":4864,"thread_start_us":126,"threads_started":1}
I20260619 16:44:07.454799 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:07.433167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60685 (local address 127.0.98.3:45541)
0619 16:44:07.433845 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:07.433869 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:07.433917 (+ 48us) server_negotiation.cc:407] Connection header received
0619 16:44:07.434787 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:07.434794 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:07.435183 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:07.435592 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:07.438261 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:07.440038 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:07.441955 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:07.442809 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:07.443018 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:07.447200 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:07.447208 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:07.447224 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:07.447269 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:07.449674 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:07.450489 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:07.450494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:07.450502 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:07.450649 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:07.451191 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:07.451198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:07.451203 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:07.452057 (+ 854us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:07.452225 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:07.454125 (+ 1900us) server_negotiation.cc:299] Negotiation successful
0619 16:44:07.454558 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":160,"threads_started":1}
W20260619 16:44:07.456183 15288 leader_election.cc:341] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae08116dded4430bb72b9c0a76b69d8c (127.0.98.2:43683): Not found: Tablet not found: 2e0f698e48d847829bb2e669f887aa81
W20260619 16:44:07.457600 15289 leader_election.cc:341] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6791ec79dd4d44b084a2221643274df6 (127.0.98.3:45541): Not found: Tablet not found: 2e0f698e48d847829bb2e669f887aa81
I20260619 16:44:07.457818 15289 leader_election.cc:302] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2e3597861584b1ba48bd52fe0489942; no voters: 6791ec79dd4d44b084a2221643274df6, ae08116dded4430bb72b9c0a76b69d8c
I20260619 16:44:07.458271 15674 raft_consensus.cc:2726] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:44:07.924731 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:07.924385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:59895)
0619 16:44:07.924558 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:07.924634 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 16:44:08.073227 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:08.072630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:44991)
0619 16:44:08.073026 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:08.073080 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":148,"threads_started":1}
I20260619 16:44:08.269482 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:08.268968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:45265)
0619 16:44:08.269305 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:08.269373 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":103,"threads_started":1}
I20260619 16:44:08.926339 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:08.925764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:58731)
0619 16:44:08.926116 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:08.926195 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":100,"threads_started":1}
I20260619 16:44:09.075176 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:09.074459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:60559)
0619 16:44:09.074876 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:09.074985 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":90,"threads_started":1}
I20260619 16:44:09.252792 15683 raft_consensus.cc:482] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:44:09.252988 15683 raft_consensus.cc:504] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6791ec79dd4d44b084a2221643274df6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45541 } } peers { permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 } } peers { permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 } }
I20260619 16:44:09.253464 15683 leader_election.cc:288] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6791ec79dd4d44b084a2221643274df6 (127.0.98.3:45541), ae08116dded4430bb72b9c0a76b69d8c (127.0.98.2:43683)
W20260619 16:44:09.254612 15288 leader_election.cc:341] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae08116dded4430bb72b9c0a76b69d8c (127.0.98.2:43683): Not found: Tablet not found: 2e0f698e48d847829bb2e669f887aa81
W20260619 16:44:09.254777 15289 leader_election.cc:341] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6791ec79dd4d44b084a2221643274df6 (127.0.98.3:45541): Not found: Tablet not found: 2e0f698e48d847829bb2e669f887aa81
I20260619 16:44:09.254930 15289 leader_election.cc:302] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2e3597861584b1ba48bd52fe0489942; no voters: 6791ec79dd4d44b084a2221643274df6, ae08116dded4430bb72b9c0a76b69d8c
I20260619 16:44:09.255143 15683 raft_consensus.cc:2726] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:44:09.271194 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:09.270522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:53783)
0619 16:44:09.270873 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:09.270982 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":101,"threads_started":1}
I20260619 16:44:09.927919 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:09.927340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:45967)
0619 16:44:09.927648 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:09.927790 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":84,"threads_started":1}
I20260619 16:44:10.076920 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:10.076348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:55573)
0619 16:44:10.076693 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:10.076782 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":85,"threads_started":1}
I20260619 16:44:10.273000 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:10.272357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:40881)
0619 16:44:10.272741 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:10.272865 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":97,"threads_started":1}
I20260619 16:44:10.929858 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:10.929226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:39619)
0619 16:44:10.929663 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:10.929749 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":114,"threads_started":1}
I20260619 16:44:11.078701 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:11.078082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:36357)
0619 16:44:11.078501 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:11.078589 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":121,"threads_started":1}
I20260619 16:44:11.274744 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:11.274188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:57691)
0619 16:44:11.274537 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:11.274625 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":99,"threads_started":1}
I20260619 16:44:11.881189 15691 raft_consensus.cc:482] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:44:11.881394 15691 raft_consensus.cc:504] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6791ec79dd4d44b084a2221643274df6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45541 } } peers { permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 } } peers { permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 } }
I20260619 16:44:11.882056 15691 leader_election.cc:288] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6791ec79dd4d44b084a2221643274df6 (127.0.98.3:45541), ae08116dded4430bb72b9c0a76b69d8c (127.0.98.2:43683)
W20260619 16:44:11.883255 15289 leader_election.cc:341] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6791ec79dd4d44b084a2221643274df6 (127.0.98.3:45541): Not found: Tablet not found: 2e0f698e48d847829bb2e669f887aa81
W20260619 16:44:11.883440 15288 leader_election.cc:341] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae08116dded4430bb72b9c0a76b69d8c (127.0.98.2:43683): Not found: Tablet not found: 2e0f698e48d847829bb2e669f887aa81
I20260619 16:44:11.883648 15288 leader_election.cc:302] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2e3597861584b1ba48bd52fe0489942; no voters: 6791ec79dd4d44b084a2221643274df6, ae08116dded4430bb72b9c0a76b69d8c
I20260619 16:44:11.884037 15691 raft_consensus.cc:2726] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:44:11.931831 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:11.931091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:45987)
0619 16:44:11.931521 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:11.931651 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":156,"threads_started":1}
I20260619 16:44:12.080439 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:12.079854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:37643)
0619 16:44:12.080235 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:12.080304 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":147,"threads_started":1}
I20260619 16:44:12.276350 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:12.275775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:52997)
0619 16:44:12.276131 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:12.276215 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":107,"threads_started":1}
I20260619 16:44:12.933369 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:12.932833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:37997)
0619 16:44:12.933178 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:12.933263 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":98,"threads_started":1}
I20260619 16:44:13.082228 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:13.081616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:47619)
0619 16:44:13.082037 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:13.082110 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":97,"threads_started":1}
I20260619 16:44:13.278240 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:13.277467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:44399)
0619 16:44:13.278002 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:13.278095 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":98,"threads_started":1}
I20260619 16:44:13.935322 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:13.934640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:58243)
0619 16:44:13.935063 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:13.935185 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":110,"threads_started":1}
I20260619 16:44:14.084219 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:14.083532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:37899)
0619 16:44:14.084009 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:14.084096 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":148,"threads_started":1}
I20260619 16:44:14.144410 15700 raft_consensus.cc:482] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:44:14.144582 15700 raft_consensus.cc:504] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6791ec79dd4d44b084a2221643274df6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45541 } } peers { permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 } } peers { permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 } }
I20260619 16:44:14.145078 15700 leader_election.cc:288] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6791ec79dd4d44b084a2221643274df6 (127.0.98.3:45541), ae08116dded4430bb72b9c0a76b69d8c (127.0.98.2:43683)
W20260619 16:44:14.146270 15288 leader_election.cc:341] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae08116dded4430bb72b9c0a76b69d8c (127.0.98.2:43683): Not found: Tablet not found: 2e0f698e48d847829bb2e669f887aa81
W20260619 16:44:14.146435 15289 leader_election.cc:341] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6791ec79dd4d44b084a2221643274df6 (127.0.98.3:45541): Not found: Tablet not found: 2e0f698e48d847829bb2e669f887aa81
I20260619 16:44:14.146498 15289 leader_election.cc:302] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2e3597861584b1ba48bd52fe0489942; no voters: 6791ec79dd4d44b084a2221643274df6, ae08116dded4430bb72b9c0a76b69d8c
I20260619 16:44:14.146698 15700 raft_consensus.cc:2726] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:44:14.280092 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:14.279419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:36707)
0619 16:44:14.279852 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:14.279941 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":217,"threads_started":1}
I20260619 16:44:14.937071 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:14.936462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:34439)
0619 16:44:14.936840 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:14.936925 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":97,"threads_started":1}
I20260619 16:44:15.086045 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:15.085431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:59823)
0619 16:44:15.085853 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:15.085940 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":123,"threads_started":1}
I20260619 16:44:15.281998 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:15.281222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:41783)
0619 16:44:15.281649 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:15.281851 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":126,"threads_started":1}
I20260619 16:44:15.938938 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:15.938306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:57569)
0619 16:44:15.938711 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:15.938807 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":115,"threads_started":1}
I20260619 16:44:16.087807 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:16.087126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:43945)
0619 16:44:16.087515 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:16.087606 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":86,"threads_started":1}
I20260619 16:44:16.283871 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:16.283243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:53557)
0619 16:44:16.283588 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:16.283728 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":84,"threads_started":1}
I20260619 16:44:16.940913 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:16.940222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:60373)
0619 16:44:16.940643 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:16.940749 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":128,"threads_started":1}
I20260619 16:44:17.089494 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:17.088826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:45523)
0619 16:44:17.089260 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:17.089358 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":122,"threads_started":1}
I20260619 16:44:17.285400 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:17.284840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:35465)
0619 16:44:17.285166 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:17.285239 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":95,"threads_started":1}
I20260619 16:44:17.806993 15711 raft_consensus.cc:482] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:44:17.807192 15711 raft_consensus.cc:504] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6791ec79dd4d44b084a2221643274df6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45541 } } peers { permanent_uuid: "ae08116dded4430bb72b9c0a76b69d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43683 } } peers { permanent_uuid: "c2e3597861584b1ba48bd52fe0489942" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45375 } }
I20260619 16:44:17.807783 15711 leader_election.cc:288] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6791ec79dd4d44b084a2221643274df6 (127.0.98.3:45541), ae08116dded4430bb72b9c0a76b69d8c (127.0.98.2:43683)
W20260619 16:44:17.808804 15289 leader_election.cc:341] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6791ec79dd4d44b084a2221643274df6 (127.0.98.3:45541): Not found: Tablet not found: 2e0f698e48d847829bb2e669f887aa81
W20260619 16:44:17.808982 15288 leader_election.cc:341] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae08116dded4430bb72b9c0a76b69d8c (127.0.98.2:43683): Not found: Tablet not found: 2e0f698e48d847829bb2e669f887aa81
I20260619 16:44:17.809119 15288 leader_election.cc:302] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2e3597861584b1ba48bd52fe0489942; no voters: 6791ec79dd4d44b084a2221643274df6, ae08116dded4430bb72b9c0a76b69d8c
I20260619 16:44:17.809366 15711 raft_consensus.cc:2726] T 2e0f698e48d847829bb2e669f887aa81 P c2e3597861584b1ba48bd52fe0489942 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:44:17.942811 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:17.942147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:48311)
0619 16:44:17.942574 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:17.942672 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":94,"threads_started":1}
I20260619 16:44:18.091501 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:18.090882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:40123)
0619 16:44:18.091260 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:18.091360 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":133,"threads_started":1}
I20260619 16:44:18.287374 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:18.286681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:33155)
0619 16:44:18.287159 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:18.287242 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":151,"threads_started":1}
I20260619 16:44:18.944723 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:18.944113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:46275)
0619 16:44:18.944487 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:18.944613 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":105,"threads_started":1}
I20260619 16:44:19.093400 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:19.092792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:53327)
0619 16:44:19.093192 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:19.093283 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":106,"threads_started":1}
I20260619 16:44:19.289095 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:19.288506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:52827)
0619 16:44:19.288885 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:19.288970 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":100,"threads_started":1}
I20260619 16:44:19.946424 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:19.945807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:34629)
0619 16:44:19.946182 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:19.946282 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":94,"threads_started":1}
I20260619 16:44:20.095256 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:20.094515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:44239)
0619 16:44:20.095044 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:20.095130 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":227,"threads_started":1}
I20260619 16:44:20.290719 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:20.290256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:54777)
0619 16:44:20.290549 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:20.290619 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":122,"threads_started":1}
I20260619 16:44:20.948632 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:20.947923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:38643)
0619 16:44:20.948392 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:20.948503 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":118,"threads_started":1}
I20260619 16:44:21.097177 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:21.096679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:53267)
0619 16:44:21.096990 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:21.097072 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":131,"threads_started":1}
I20260619 16:44:21.268131 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 15237
I20260619 16:44:21.274978 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 15404
I20260619 16:44:21.282708 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 15537
2026-06-19T16:44:21Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19609 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:44:21 dist-test-slave-03cd krb5kdc[15725](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:44:21 dist-test-slave-03cd krb5kdc[15725](info): set up 2 sockets
Jun 19 16:44:21 dist-test-slave-03cd krb5kdc[15725](info): commencing operation
krb5kdc: starting...
W20260619 16:44:22.368232 392 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:44:22 dist-test-slave-03cd krb5kdc[15725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:44:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:44:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:44:22.503201 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46833
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:44:22.620369 15738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:22.620710 15738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:22.620755 15738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:22.624924 15738 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 16:44:22.624990 15738 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 16:44:22.625006 15738 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 16:44:22.625020 15738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:44:22.625039 15738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:22.625058 15738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:22.625074 15738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:44:22.625090 15738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:44:22.632176 15738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:44:22.633361 15738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:22.634426 15738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:22.644119 15743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:22.644138 15744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:22.644701 15746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:22.644781 15738 server_base.cc:1034] running on GCE node
I20260619 16:44:22.645463 15738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:22.646714 15738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:22.647899 15738 hybrid_clock.cc:648] HybridClock initialized: now 1781887462647849 us; error 67 us; skew 500 ppm
Jun 19 16:44:22 dist-test-slave-03cd krb5kdc[15725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:22.650892 15738 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:44:22.651993 15738 webserver.cc:460] Webserver started at http://127.0.98.62:43313/ using document root <none> and password file <none>
I20260619 16:44:22.652565 15738 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:22.652627 15738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:22.652806 15738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:44:22.654923 15738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "e90ee4f9a07e4d23b0b9a9906cb04aad"
format_stamp: "Formatted at 2026-06-19 16:44:22 on dist-test-slave-03cd"
server_key: "ca9c8c19d293685c606c11c15c9f9c1b"
server_key_iv: "587c2dc0fc606558e52d19f272a98a4f"
server_key_version: "encryptionkey@0"
I20260619 16:44:22.655486 15738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "e90ee4f9a07e4d23b0b9a9906cb04aad"
format_stamp: "Formatted at 2026-06-19 16:44:22 on dist-test-slave-03cd"
server_key: "ca9c8c19d293685c606c11c15c9f9c1b"
server_key_iv: "587c2dc0fc606558e52d19f272a98a4f"
server_key_version: "encryptionkey@0"
I20260619 16:44:22.659536 15738 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 16:44:22.662319 15753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:22.663983 15738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:44:22.664170 15738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "e90ee4f9a07e4d23b0b9a9906cb04aad"
format_stamp: "Formatted at 2026-06-19 16:44:22 on dist-test-slave-03cd"
server_key: "ca9c8c19d293685c606c11c15c9f9c1b"
server_key_iv: "587c2dc0fc606558e52d19f272a98a4f"
server_key_version: "encryptionkey@0"
I20260619 16:44:22.664321 15738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:22.679605 15738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:22.683272 15738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:22.683532 15738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:22.691075 15738 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46833
I20260619 16:44:22.691382 15805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46833 every 8 connection(s)
I20260619 16:44:22.692209 15738 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:44:22.695227 15806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:22.700402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 15738
I20260619 16:44:22.700551 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:44:22.700831 392 external_mini_cluster.cc:1428] Setting key e0b6a633f8b942764a463beb76b5b631
I20260619 16:44:22.701608 15806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad: Bootstrap starting.
I20260619 16:44:22.704269 15806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:22.705082 15806 log.cc:824] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad: Log is configured to *not* fsync() on all Append() calls
I20260619 16:44:22.707494 15806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad: No bootstrap required, opened a new log
Jun 19 16:44:22 dist-test-slave-03cd krb5kdc[15725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:44:22.712411 15806 raft_consensus.cc:348] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90ee4f9a07e4d23b0b9a9906cb04aad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46833 } }
I20260619 16:44:22.712754 15806 raft_consensus.cc:374] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:22.712855 15806 raft_consensus.cc:729] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e90ee4f9a07e4d23b0b9a9906cb04aad, State: Initialized, Role: FOLLOWER
I20260619 16:44:22.713495 15806 consensus_queue.cc:260] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90ee4f9a07e4d23b0b9a9906cb04aad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46833 } }
I20260619 16:44:22.713690 15806 raft_consensus.cc:388] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:44:22.713776 15806 raft_consensus.cc:482] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:44:22.713905 15806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:22.715176 15806 raft_consensus.cc:504] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90ee4f9a07e4d23b0b9a9906cb04aad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46833 } }
I20260619 16:44:22.715615 15806 leader_election.cc:302] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e90ee4f9a07e4d23b0b9a9906cb04aad; no voters:
I20260619 16:44:22.716039 15806 leader_election.cc:288] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:44:22.716181 15811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:44:22.716445 15811 raft_consensus.cc:686] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [term 1 LEADER]: Becoming Leader. State: Replica: e90ee4f9a07e4d23b0b9a9906cb04aad, State: Running, Role: LEADER
I20260619 16:44:22.716825 15811 consensus_queue.cc:237] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90ee4f9a07e4d23b0b9a9906cb04aad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46833 } }
I20260619 16:44:22.717511 15806 sys_catalog.cc:564] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:44:22.718590 15813 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [sys.catalog]: SysCatalogTable state changed. Reason: New leader e90ee4f9a07e4d23b0b9a9906cb04aad. Latest consensus state: current_term: 1 leader_uuid: "e90ee4f9a07e4d23b0b9a9906cb04aad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90ee4f9a07e4d23b0b9a9906cb04aad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46833 } } }
I20260619 16:44:22.718653 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:22.702376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37182 (local address 127.0.98.62:46833)
0619 16:44:22.702882 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:22.702898 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:22.702934 (+ 36us) server_negotiation.cc:407] Connection header received
0619 16:44:22.703643 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:22.703661 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:22.704101 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:22.704456 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:22.705357 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:22.707056 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:22.708110 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:22.708831 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:22.709008 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:22.712202 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:22.712209 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:22.712228 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:22.712267 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:22.714511 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:22.715031 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:22.715041 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:22.715050 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:22.715217 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:22.715551 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:22.715554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:22.715555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:22.716204 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:22.716413 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:22.717215 (+ 802us) server_negotiation.cc:299] Negotiation successful
0619 16:44:22.717543 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":151,"threads_started":1}
I20260619 16:44:22.718743 15813 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [sys.catalog]: This master's current role is: LEADER
I20260619 16:44:22.719030 15812 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e90ee4f9a07e4d23b0b9a9906cb04aad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90ee4f9a07e4d23b0b9a9906cb04aad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46833 } } }
I20260619 16:44:22.719108 15812 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [sys.catalog]: This master's current role is: LEADER
I20260619 16:44:22.719130 15820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:44:22.723832 15820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:44:22.730353 15820 catalog_manager.cc:1347] Generated new cluster ID: 2ad0be05d5354c1f96ef6fd1f980397a
I20260619 16:44:22.730463 15820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:44:22.747038 15820 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:44:22.748498 15820 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:44:22.755333 15820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad: Generated new TSK 0
I20260619 16:44:22.756157 15820 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:44:22.800127 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46833
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:44:22.924057 15832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:22.924458 15832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:22.924588 15832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:22.929255 15832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:22.929390 15832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:22.929497 15832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:44:22.935451 15832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:44:22.936758 15832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:22.937851 15832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:22.948194 15840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:22.948201 15837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:22.948203 15838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:22.948319 15832 server_base.cc:1034] running on GCE node
I20260619 16:44:22.948915 15832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:22.949640 15832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:22.950863 15832 hybrid_clock.cc:648] HybridClock initialized: now 1781887462950848 us; error 29 us; skew 500 ppm
Jun 19 16:44:22 dist-test-slave-03cd krb5kdc[15725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:22.953959 15832 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:44:22.954885 15832 webserver.cc:460] Webserver started at http://127.0.98.1:34003/ using document root <none> and password file <none>
I20260619 16:44:22.955449 15832 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:22.955509 15832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:22.955750 15832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:44:22.957891 15832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/data/instance:
uuid: "869d5f0889b5494d8fb502c4b810df76"
format_stamp: "Formatted at 2026-06-19 16:44:22 on dist-test-slave-03cd"
server_key: "57fb264417f98af538e63f3865d456eb"
server_key_iv: "bcdb807774b4ffd66a31df8714c9f22b"
server_key_version: "encryptionkey@0"
I20260619 16:44:22.958426 15832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/wal/instance:
uuid: "869d5f0889b5494d8fb502c4b810df76"
format_stamp: "Formatted at 2026-06-19 16:44:22 on dist-test-slave-03cd"
server_key: "57fb264417f98af538e63f3865d456eb"
server_key_iv: "bcdb807774b4ffd66a31df8714c9f22b"
server_key_version: "encryptionkey@0"
I20260619 16:44:22.962672 15832 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260619 16:44:22.965411 15847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:22.966723 15832 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:44:22.966846 15832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "869d5f0889b5494d8fb502c4b810df76"
format_stamp: "Formatted at 2026-06-19 16:44:22 on dist-test-slave-03cd"
server_key: "57fb264417f98af538e63f3865d456eb"
server_key_iv: "bcdb807774b4ffd66a31df8714c9f22b"
server_key_version: "encryptionkey@0"
I20260619 16:44:22.967018 15832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:22.980438 15832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:22.984031 15832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:22.984293 15832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:22.984992 15832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:44:22.985905 15832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:44:22.985954 15832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:22.985998 15832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:44:22.986071 15832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:22.996260 15832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44505
I20260619 16:44:22.996279 15960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44505 every 8 connection(s)
I20260619 16:44:22.997257 15832 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:44:22.998142 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 15832
I20260619 16:44:22.998276 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:44:22.998569 392 external_mini_cluster.cc:1428] Setting key 7dd10c6e3dd3a0df12cc15124ffe7cc1
Jun 19 16:44:23 dist-test-slave-03cd krb5kdc[15725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 16:44:23.015727 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:22.999340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35259 (local address 127.0.98.62:46833)
0619 16:44:22.999487 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:22.999492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:23.000537 (+ 1045us) server_negotiation.cc:407] Connection header received
0619 16:44:23.002264 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:23.002267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:23.002321 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:23.002397 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:23.004057 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.005125 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.006507 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.006789 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.006896 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:23.010849 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:23.010856 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:23.010859 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:23.010899 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:23.012755 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:23.013493 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:23.013496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:23.013497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:23.013588 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:23.014238 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:23.014240 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:23.014241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:23.014520 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:23.014629 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:23.015262 (+ 633us) server_negotiation.cc:299] Negotiation successful
0619 16:44:23.015432 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:44:23.016662 15963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:22.999838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.1:35259)
0619 16:44:23.000367 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:23.000422 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:23.001961 (+ 1539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:23.002614 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:23.002624 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:23.003192 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:23.003820 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:23.003837 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.005324 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.005327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:23.006341 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:23.006352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.007021 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.007028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:23.007336 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:23.008518 (+ 1182us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:23.008544 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:23.010541 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:23.012940 (+ 2399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:23.012947 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:23.012964 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:23.013365 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:23.013712 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:23.013714 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:23.013715 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:23.014068 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:23.014681 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:23.014687 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:23.014969 (+ 282us) client_negotiation.cc:769] Sending connection context
0619 16:44:23.015141 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 16:44:23.015439 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":163,"threads_started":1}
I20260619 16:44:23.018307 15961 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46833
I20260619 16:44:23.018787 15961 heartbeater.cc:464] Registering TS with master...
I20260619 16:44:23.019482 15961 heartbeater.cc:511] Master 127.0.98.62:46833 requested a full tablet report, sending...
I20260619 16:44:23.021940 15770 ts_manager.cc:194] Registered new tserver with Master: 869d5f0889b5494d8fb502c4b810df76 (127.0.98.1:44505)
I20260619 16:44:23.023795 15770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:44:23.036336 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46833
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:44:23.159104 15966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:23.159456 15966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:23.159507 15966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:23.164494 15966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:23.164582 15966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:23.164654 15966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:44:23.172479 15966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:44:23.173806 15966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:23.174916 15966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:23.185403 15974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:23.185392 15971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:23.185395 15972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:23.186033 15966 server_base.cc:1034] running on GCE node
I20260619 16:44:23.186580 15966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:23.187320 15966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:23.188529 15966 hybrid_clock.cc:648] HybridClock initialized: now 1781887463188508 us; error 58 us; skew 500 ppm
Jun 19 16:44:23 dist-test-slave-03cd krb5kdc[15725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:23.191596 15966 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:44:23.192649 15966 webserver.cc:460] Webserver started at http://127.0.98.2:44013/ using document root <none> and password file <none>
I20260619 16:44:23.193292 15966 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:23.193387 15966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:23.193634 15966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:44:23.195883 15966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/data/instance:
uuid: "53176a8d1367494f9eeb38acefecfa0c"
format_stamp: "Formatted at 2026-06-19 16:44:23 on dist-test-slave-03cd"
server_key: "18caedd2a847cdea4ba46431788b9649"
server_key_iv: "a6743a74054908d353ad32c88c9f0e31"
server_key_version: "encryptionkey@0"
I20260619 16:44:23.196497 15966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/wal/instance:
uuid: "53176a8d1367494f9eeb38acefecfa0c"
format_stamp: "Formatted at 2026-06-19 16:44:23 on dist-test-slave-03cd"
server_key: "18caedd2a847cdea4ba46431788b9649"
server_key_iv: "a6743a74054908d353ad32c88c9f0e31"
server_key_version: "encryptionkey@0"
I20260619 16:44:23.200536 15966 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 16:44:23.203279 15981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:23.204849 15966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:44:23.205021 15966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "53176a8d1367494f9eeb38acefecfa0c"
format_stamp: "Formatted at 2026-06-19 16:44:23 on dist-test-slave-03cd"
server_key: "18caedd2a847cdea4ba46431788b9649"
server_key_iv: "a6743a74054908d353ad32c88c9f0e31"
server_key_version: "encryptionkey@0"
I20260619 16:44:23.205159 15966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:23.223431 15966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:23.227252 15966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:23.227487 15966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:23.228204 15966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:44:23.229116 15966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:44:23.229195 15966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:23.229270 15966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:44:23.229322 15966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:23.238957 15966 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32803
I20260619 16:44:23.238994 16094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32803 every 8 connection(s)
I20260619 16:44:23.240064 15966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:44:23.244127 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 15966
I20260619 16:44:23.244239 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:44:23.244521 392 external_mini_cluster.cc:1428] Setting key 32e0c7f8826de7c0618e4e1b52a1bc63
Jun 19 16:44:23 dist-test-slave-03cd krb5kdc[15725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:44:23.257270 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:23.242231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59807 (local address 127.0.98.62:46833)
0619 16:44:23.242398 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:23.242403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:23.243348 (+ 945us) server_negotiation.cc:407] Connection header received
0619 16:44:23.244938 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:23.244942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:23.245003 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:23.245083 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:23.246675 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.247539 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.249004 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.249429 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.249569 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:23.252449 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:23.252455 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:23.252461 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:23.252497 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:23.254414 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:23.255187 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:23.255195 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:23.255199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:23.255316 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:23.255923 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:23.255926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:23.255927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:23.256159 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:23.256248 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:23.257002 (+ 754us) server_negotiation.cc:299] Negotiation successful
0619 16:44:23.257130 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:44:23.258276 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:23.242685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.2:59807)
0619 16:44:23.243176 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:23.243223 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:23.244692 (+ 1469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:23.245298 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:23.245334 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:23.245840 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:23.246512 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:23.246528 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.247765 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.247768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:23.248859 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:23.248870 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.249598 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.249601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:23.249769 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:23.250448 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:23.250466 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:23.252240 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:23.254658 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:23.254669 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:23.254691 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:23.255021 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:23.255484 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:23.255490 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:23.255493 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:23.255782 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:23.256284 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:23.256290 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:23.256672 (+ 382us) client_negotiation.cc:769] Sending connection context
0619 16:44:23.256911 (+ 239us) client_negotiation.cc:240] Negotiation successful
0619 16:44:23.257257 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 16:44:23.259949 16095 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46833
I20260619 16:44:23.260282 16095 heartbeater.cc:464] Registering TS with master...
I20260619 16:44:23.260881 16095 heartbeater.cc:511] Master 127.0.98.62:46833 requested a full tablet report, sending...
I20260619 16:44:23.262387 15770 ts_manager.cc:194] Registered new tserver with Master: 53176a8d1367494f9eeb38acefecfa0c (127.0.98.2:32803)
I20260619 16:44:23.263336 15770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59807
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:44:23.284482 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46833
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:44:23.404070 16100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:23.404464 16100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:23.404513 16100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:23.408888 16100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:23.408967 16100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:23.409062 16100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:44:23.414846 16100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:44:23.416124 16100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:23.417210 16100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:44:23.427878 16100 server_base.cc:1034] running on GCE node
W20260619 16:44:23.427829 16108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:23.427829 16105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:23.427829 16106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:23.428610 16100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:23.429265 16100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:23.430500 16100 hybrid_clock.cc:648] HybridClock initialized: now 1781887463430484 us; error 28 us; skew 500 ppm
Jun 19 16:44:23 dist-test-slave-03cd krb5kdc[15725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:23.433434 16100 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:44:23.434424 16100 webserver.cc:460] Webserver started at http://127.0.98.3:45957/ using document root <none> and password file <none>
I20260619 16:44:23.435107 16100 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:23.435204 16100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:23.435446 16100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:44:23.437741 16100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/data/instance:
uuid: "18ec02b6990242519665eb24c9fa5f14"
format_stamp: "Formatted at 2026-06-19 16:44:23 on dist-test-slave-03cd"
server_key: "0907382f51814a0c728ce713b280cb08"
server_key_iv: "ff78012f373e85958ca966a514180804"
server_key_version: "encryptionkey@0"
I20260619 16:44:23.438354 16100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/wal/instance:
uuid: "18ec02b6990242519665eb24c9fa5f14"
format_stamp: "Formatted at 2026-06-19 16:44:23 on dist-test-slave-03cd"
server_key: "0907382f51814a0c728ce713b280cb08"
server_key_iv: "ff78012f373e85958ca966a514180804"
server_key_version: "encryptionkey@0"
I20260619 16:44:23.442559 16100 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:44:23.445698 16115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:23.447104 16100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 16:44:23.447294 16100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "18ec02b6990242519665eb24c9fa5f14"
format_stamp: "Formatted at 2026-06-19 16:44:23 on dist-test-slave-03cd"
server_key: "0907382f51814a0c728ce713b280cb08"
server_key_iv: "ff78012f373e85958ca966a514180804"
server_key_version: "encryptionkey@0"
I20260619 16:44:23.447439 16100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:23.463240 16100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:23.467026 16100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:23.467262 16100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:23.468024 16100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:44:23.469048 16100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:44:23.469125 16100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:23.469200 16100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:44:23.469244 16100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:23.478547 16100 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43281
I20260619 16:44:23.478580 16228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43281 every 8 connection(s)
I20260619 16:44:23.479509 16100 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:44:23.486009 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 16100
I20260619 16:44:23.486145 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:44:23.486413 392 external_mini_cluster.cc:1428] Setting key 232d12057bab602658a6cd3998aae122
Jun 19 16:44:23 dist-test-slave-03cd krb5kdc[15725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:44:23.493971 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:23.481484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59773 (local address 127.0.98.62:46833)
0619 16:44:23.481654 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:23.481658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:23.483035 (+ 1377us) server_negotiation.cc:407] Connection header received
0619 16:44:23.484026 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:23.484029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:23.484078 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:23.484148 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:23.485295 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.485899 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.487247 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.487571 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.487742 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:23.490341 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:23.490345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:23.490347 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:23.490372 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:23.491832 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:23.492341 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:23.492344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:23.492346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:23.492431 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:23.492815 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:23.492818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:23.492820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:23.493020 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:23.493113 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:23.493717 (+ 604us) server_negotiation.cc:299] Negotiation successful
0619 16:44:23.493830 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:44:23.494824 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:23.481868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.3:59773)
0619 16:44:23.482871 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:23.482915 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:23.483816 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:23.484305 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:23.484311 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:23.484686 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:23.485167 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:23.485180 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.486041 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.486043 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:23.487123 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:23.487133 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.487774 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.487778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:23.487975 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:23.488692 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:23.488711 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:23.490186 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:23.491943 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:23.491948 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:23.491960 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:23.492227 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:23.492574 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:23.492576 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:23.492577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:23.492710 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:23.493147 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:23.493153 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:23.493438 (+ 285us) client_negotiation.cc:769] Sending connection context
0619 16:44:23.493617 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 16:44:23.493896 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"thread_start_us":646,"threads_started":1}
I20260619 16:44:23.496064 16229 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46833
I20260619 16:44:23.496320 16229 heartbeater.cc:464] Registering TS with master...
I20260619 16:44:23.496865 16229 heartbeater.cc:511] Master 127.0.98.62:46833 requested a full tablet report, sending...
I20260619 16:44:23.497991 15770 ts_manager.cc:194] Registered new tserver with Master: 18ec02b6990242519665eb24c9fa5f14 (127.0.98.3:43281)
I20260619 16:44:23.498657 15770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59773
I20260619 16:44:23.500617 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:44:23.510448 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:23.502931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37194 (local address 127.0.98.62:46833)
0619 16:44:23.503103 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:23.503107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:23.503198 (+ 91us) server_negotiation.cc:407] Connection header received
0619 16:44:23.503289 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:23.503291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:23.503331 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:23.503399 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:23.504191 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.504736 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.505614 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.505859 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.505942 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:23.507335 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:23.507338 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:23.507340 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:23.507363 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:23.508799 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:23.509167 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:23.509170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:23.509171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:23.509242 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:23.509461 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:23.509463 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:23.509464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:23.509735 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:23.509883 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:23.510154 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 16:44:23.510275 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 16:44:23.513429 15770 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:44:23.516600 15770 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:44:23.531298 16240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:23.525314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44505 (local address 127.0.0.1:60526)
0619 16:44:23.525960 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:23.526166 (+ 206us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:23.526248 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:23.527277 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:23.527279 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:23.527293 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:23.527364 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:23.527368 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.528526 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.528532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:23.529771 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:23.529780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.530648 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.530652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:23.530836 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:23.530891 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 16:44:23.531006 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 16:44:23.531126 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":26,"spinlock_wait_cycles":15104,"thread_start_us":103,"threads_started":1}
I20260619 16:44:23.531335 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:23.526004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60526 (local address 127.0.98.1:44505)
0619 16:44:23.526824 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:23.526830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:23.526854 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:44:23.526929 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:23.526933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:23.527096 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:23.527286 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:23.527482 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.528420 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.529906 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.530513 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.530648 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:23.530819 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:23.531059 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 16:44:23.531177 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":90,"threads_started":1}
I20260619 16:44:23.531527 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:23.525215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43281 (local address 127.0.0.1:43390)
0619 16:44:23.525961 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:23.526018 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:23.526429 (+ 411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:23.526966 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:23.526970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:23.526991 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:23.527102 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:23.527107 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.528586 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.528590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:23.529907 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:23.529917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.531131 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.531135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:23.531216 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:23.531239 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:44:23.531309 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 16:44:23.531392 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":57,"thread_start_us":90,"threads_started":1}
I20260619 16:44:23.531877 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:23.525329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43390 (local address 127.0.98.3:43281)
0619 16:44:23.526461 (+ 1132us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:23.526477 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:23.526502 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:44:23.526577 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:23.526582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:23.526803 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:23.526965 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:23.527241 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.528430 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.530035 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.530828 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.530932 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:23.531072 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:23.531603 (+ 531us) server_negotiation.cc:299] Negotiation successful
0619 16:44:23.531751 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1021,"thread_start_us":85,"threads_started":1}
I20260619 16:44:23.532511 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:23.525591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32803 (local address 127.0.0.1:35770)
0619 16:44:23.526036 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:23.526051 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:23.526162 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:23.527984 (+ 1822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:23.527987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:23.528005 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:23.528110 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:23.528115 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.529533 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.529536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:23.530786 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:23.530799 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.532116 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.532119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:23.532223 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:23.532245 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 16:44:23.532315 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 16:44:23.532383 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":78,"thread_start_us":94,"threads_started":1}
I20260619 16:44:23.533095 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:23.526378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35770 (local address 127.0.98.2:32803)
0619 16:44:23.527551 (+ 1173us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:23.527562 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:23.527582 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:44:23.527650 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:23.527678 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:23.527843 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:23.527986 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:23.528257 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.529383 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.530895 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.531807 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.531919 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:23.532051 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:23.532859 (+ 808us) server_negotiation.cc:299] Negotiation successful
0619 16:44:23.532960 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1079,"thread_start_us":93,"threads_started":1}
I20260619 16:44:23.534817 15895 tablet_service.cc:1467] Processing CreateTablet for tablet c0ec67fc1761441196d3d861225d009a (DEFAULT_TABLE table=test-table [id=1238014d302f40b2aa84b2e4507cbfdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:44:23.534860 16163 tablet_service.cc:1467] Processing CreateTablet for tablet c0ec67fc1761441196d3d861225d009a (DEFAULT_TABLE table=test-table [id=1238014d302f40b2aa84b2e4507cbfdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:44:23.535953 15895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0ec67fc1761441196d3d861225d009a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:23.535969 16163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0ec67fc1761441196d3d861225d009a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:23.535652 16029 tablet_service.cc:1467] Processing CreateTablet for tablet c0ec67fc1761441196d3d861225d009a (DEFAULT_TABLE table=test-table [id=1238014d302f40b2aa84b2e4507cbfdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:44:23.536664 16029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0ec67fc1761441196d3d861225d009a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:23.542639 16246 tablet_bootstrap.cc:492] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76: Bootstrap starting.
I20260619 16:44:23.544231 16247 tablet_bootstrap.cc:492] T c0ec67fc1761441196d3d861225d009a P 18ec02b6990242519665eb24c9fa5f14: Bootstrap starting.
I20260619 16:44:23.545004 16248 tablet_bootstrap.cc:492] T c0ec67fc1761441196d3d861225d009a P 53176a8d1367494f9eeb38acefecfa0c: Bootstrap starting.
I20260619 16:44:23.545418 16246 tablet_bootstrap.cc:654] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:23.546205 16246 log.cc:824] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76: Log is configured to *not* fsync() on all Append() calls
I20260619 16:44:23.546622 16247 tablet_bootstrap.cc:654] T c0ec67fc1761441196d3d861225d009a P 18ec02b6990242519665eb24c9fa5f14: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:23.547408 16248 tablet_bootstrap.cc:654] T c0ec67fc1761441196d3d861225d009a P 53176a8d1367494f9eeb38acefecfa0c: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:23.547941 16247 log.cc:824] T c0ec67fc1761441196d3d861225d009a P 18ec02b6990242519665eb24c9fa5f14: Log is configured to *not* fsync() on all Append() calls
I20260619 16:44:23.548180 16248 log.cc:824] T c0ec67fc1761441196d3d861225d009a P 53176a8d1367494f9eeb38acefecfa0c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:44:23.548377 16246 tablet_bootstrap.cc:492] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76: No bootstrap required, opened a new log
I20260619 16:44:23.548589 16246 ts_tablet_manager.cc:1397] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 16:44:23.549889 16247 tablet_bootstrap.cc:492] T c0ec67fc1761441196d3d861225d009a P 18ec02b6990242519665eb24c9fa5f14: No bootstrap required, opened a new log
I20260619 16:44:23.550088 16247 ts_tablet_manager.cc:1397] T c0ec67fc1761441196d3d861225d009a P 18ec02b6990242519665eb24c9fa5f14: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 16:44:23.553170 16246 raft_consensus.cc:348] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 } } peers { permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 } } peers { permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } }
I20260619 16:44:23.553488 16246 raft_consensus.cc:374] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:23.553565 16246 raft_consensus.cc:729] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869d5f0889b5494d8fb502c4b810df76, State: Initialized, Role: FOLLOWER
I20260619 16:44:23.553524 16247 raft_consensus.cc:348] T c0ec67fc1761441196d3d861225d009a P 18ec02b6990242519665eb24c9fa5f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 } } peers { permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 } } peers { permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } }
I20260619 16:44:23.553864 16247 raft_consensus.cc:374] T c0ec67fc1761441196d3d861225d009a P 18ec02b6990242519665eb24c9fa5f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:23.553922 16247 raft_consensus.cc:729] T c0ec67fc1761441196d3d861225d009a P 18ec02b6990242519665eb24c9fa5f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18ec02b6990242519665eb24c9fa5f14, State: Initialized, Role: FOLLOWER
I20260619 16:44:23.554132 16246 consensus_queue.cc:260] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 } } peers { permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 } } peers { permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } }
I20260619 16:44:23.554255 16247 consensus_queue.cc:260] T c0ec67fc1761441196d3d861225d009a P 18ec02b6990242519665eb24c9fa5f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 } } peers { permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 } } peers { permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } }
I20260619 16:44:23.554940 15961 heartbeater.cc:503] Master 127.0.98.62:46833 was elected leader, sending a full tablet report...
I20260619 16:44:23.555236 16247 ts_tablet_manager.cc:1428] T c0ec67fc1761441196d3d861225d009a P 18ec02b6990242519665eb24c9fa5f14: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260619 16:44:23.555284 16246 ts_tablet_manager.cc:1428] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 16:44:23.555472 16248 tablet_bootstrap.cc:492] T c0ec67fc1761441196d3d861225d009a P 53176a8d1367494f9eeb38acefecfa0c: No bootstrap required, opened a new log
I20260619 16:44:23.555755 16248 ts_tablet_manager.cc:1397] T c0ec67fc1761441196d3d861225d009a P 53176a8d1367494f9eeb38acefecfa0c: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260619 16:44:23.555810 16229 heartbeater.cc:503] Master 127.0.98.62:46833 was elected leader, sending a full tablet report...
I20260619 16:44:23.560035 16248 raft_consensus.cc:348] T c0ec67fc1761441196d3d861225d009a P 53176a8d1367494f9eeb38acefecfa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 } } peers { permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 } } peers { permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } }
I20260619 16:44:23.560284 16248 raft_consensus.cc:374] T c0ec67fc1761441196d3d861225d009a P 53176a8d1367494f9eeb38acefecfa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:23.560328 16248 raft_consensus.cc:729] T c0ec67fc1761441196d3d861225d009a P 53176a8d1367494f9eeb38acefecfa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53176a8d1367494f9eeb38acefecfa0c, State: Initialized, Role: FOLLOWER
I20260619 16:44:23.560838 16248 consensus_queue.cc:260] T c0ec67fc1761441196d3d861225d009a P 53176a8d1367494f9eeb38acefecfa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 } } peers { permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 } } peers { permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } }
I20260619 16:44:23.561650 16095 heartbeater.cc:503] Master 127.0.98.62:46833 was elected leader, sending a full tablet report...
I20260619 16:44:23.561904 16248 ts_tablet_manager.cc:1428] T c0ec67fc1761441196d3d861225d009a P 53176a8d1367494f9eeb38acefecfa0c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 16:44:23.621737 16252 raft_consensus.cc:482] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:44:23.622020 16252 raft_consensus.cc:504] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 } } peers { permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 } } peers { permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } }
I20260619 16:44:23.623309 16252 leader_election.cc:288] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18ec02b6990242519665eb24c9fa5f14 (127.0.98.3:43281), 53176a8d1367494f9eeb38acefecfa0c (127.0.98.2:32803)
I20260619 16:44:23.628299 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:23.623853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54723 (local address 127.0.98.2:32803)
0619 16:44:23.624077 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:23.624081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:23.624354 (+ 273us) server_negotiation.cc:407] Connection header received
0619 16:44:23.624513 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:23.624517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:23.624570 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:23.624663 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:23.624986 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.625825 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.627188 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.627607 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.627800 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:23.627872 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:23.628064 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 16:44:23.628144 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260619 16:44:23.628340 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:23.623850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44663 (local address 127.0.98.3:43281)
0619 16:44:23.623977 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:23.623981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:23.624329 (+ 348us) server_negotiation.cc:407] Connection header received
0619 16:44:23.624502 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:23.624508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:23.624564 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:23.624663 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:23.625094 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.625838 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.627099 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.627634 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.627818 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:23.627892 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:23.628107 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 16:44:23.628202 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:44:23.628360 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:23.623707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32803 (local address 127.0.98.1:54723)
0619 16:44:23.624234 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:23.624247 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:23.624399 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:23.624672 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:23.624675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:23.624691 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:23.624863 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:23.624869 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.625955 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.625959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:23.627037 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:23.627052 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.627815 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.627819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:23.627919 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:23.627949 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 16:44:23.628048 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 16:44:23.628136 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":272,"thread_start_us":100,"threads_started":1}
I20260619 16:44:23.628365 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:23.623711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43281 (local address 127.0.98.1:44663)
0619 16:44:23.624189 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:23.624204 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:23.624367 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:23.624676 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:23.624680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:23.624710 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:23.624874 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:23.624880 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.625972 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.625977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:23.626945 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:23.626956 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.627826 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.627830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:23.627956 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:23.627988 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 16:44:23.628088 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 16:44:23.628173 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":3,"spinlock_wait_cycles":11520,"thread_start_us":126,"threads_started":1}
I20260619 16:44:23.629312 16183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0ec67fc1761441196d3d861225d009a" candidate_uuid: "869d5f0889b5494d8fb502c4b810df76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18ec02b6990242519665eb24c9fa5f14" is_pre_election: true
I20260619 16:44:23.629317 16049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0ec67fc1761441196d3d861225d009a" candidate_uuid: "869d5f0889b5494d8fb502c4b810df76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53176a8d1367494f9eeb38acefecfa0c" is_pre_election: true
I20260619 16:44:23.629650 16049 raft_consensus.cc:2454] T c0ec67fc1761441196d3d861225d009a P 53176a8d1367494f9eeb38acefecfa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869d5f0889b5494d8fb502c4b810df76 in term 0.
I20260619 16:44:23.629658 16183 raft_consensus.cc:2454] T c0ec67fc1761441196d3d861225d009a P 18ec02b6990242519665eb24c9fa5f14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869d5f0889b5494d8fb502c4b810df76 in term 0.
I20260619 16:44:23.630218 15851 leader_election.cc:302] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 18ec02b6990242519665eb24c9fa5f14, 53176a8d1367494f9eeb38acefecfa0c, 869d5f0889b5494d8fb502c4b810df76; no voters:
I20260619 16:44:23.630548 16252 raft_consensus.cc:2781] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:44:23.630640 16252 raft_consensus.cc:482] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:44:23.630684 16252 raft_consensus.cc:3037] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:23.631992 16252 raft_consensus.cc:504] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 } } peers { permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 } } peers { permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } }
I20260619 16:44:23.632475 16252 leader_election.cc:288] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76 [CANDIDATE]: Term 1 election: Requested vote from peers 18ec02b6990242519665eb24c9fa5f14 (127.0.98.3:43281), 53176a8d1367494f9eeb38acefecfa0c (127.0.98.2:32803)
I20260619 16:44:23.632988 16183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0ec67fc1761441196d3d861225d009a" candidate_uuid: "869d5f0889b5494d8fb502c4b810df76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18ec02b6990242519665eb24c9fa5f14"
I20260619 16:44:23.633234 16183 raft_consensus.cc:3037] T c0ec67fc1761441196d3d861225d009a P 18ec02b6990242519665eb24c9fa5f14 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:23.633386 16049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0ec67fc1761441196d3d861225d009a" candidate_uuid: "869d5f0889b5494d8fb502c4b810df76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53176a8d1367494f9eeb38acefecfa0c"
I20260619 16:44:23.633535 16049 raft_consensus.cc:3037] T c0ec67fc1761441196d3d861225d009a P 53176a8d1367494f9eeb38acefecfa0c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:23.634768 16049 raft_consensus.cc:2454] T c0ec67fc1761441196d3d861225d009a P 53176a8d1367494f9eeb38acefecfa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869d5f0889b5494d8fb502c4b810df76 in term 1.
I20260619 16:44:23.634824 16183 raft_consensus.cc:2454] T c0ec67fc1761441196d3d861225d009a P 18ec02b6990242519665eb24c9fa5f14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869d5f0889b5494d8fb502c4b810df76 in term 1.
I20260619 16:44:23.635128 15851 leader_election.cc:302] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53176a8d1367494f9eeb38acefecfa0c, 869d5f0889b5494d8fb502c4b810df76; no voters:
I20260619 16:44:23.635316 16252 raft_consensus.cc:2781] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:44:23.635541 16252 raft_consensus.cc:686] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76 [term 1 LEADER]: Becoming Leader. State: Replica: 869d5f0889b5494d8fb502c4b810df76, State: Running, Role: LEADER
I20260619 16:44:23.635943 16252 consensus_queue.cc:237] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 } } peers { permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 } } peers { permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } }
I20260619 16:44:23.639633 15769 catalog_manager.cc:5495] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76 reported cstate change: term changed from 0 to 1, leader changed from <none> to 869d5f0889b5494d8fb502c4b810df76 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "869d5f0889b5494d8fb502c4b810df76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } health_report { overall_health: UNKNOWN } } }
I20260619 16:44:23.658931 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:23.655357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60538 (local address 127.0.98.1:44505)
0619 16:44:23.655489 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:23.655492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:23.655505 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:44:23.655610 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:23.655612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:23.655741 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:23.655836 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:44:23.656078 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.656582 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.657487 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.657718 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.657796 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:23.658103 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:44:23.658537 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:44:23.658628 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:23.658758 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 16:44:23.658814 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:44:23.665388 16183 raft_consensus.cc:1261] T c0ec67fc1761441196d3d861225d009a P 18ec02b6990242519665eb24c9fa5f14 [term 1 FOLLOWER]: Refusing update from remote peer 869d5f0889b5494d8fb502c4b810df76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:44:23.665388 16049 raft_consensus.cc:1261] T c0ec67fc1761441196d3d861225d009a P 53176a8d1367494f9eeb38acefecfa0c [term 1 FOLLOWER]: Refusing update from remote peer 869d5f0889b5494d8fb502c4b810df76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:44:23.666239 16257 consensus_queue.cc:1035] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:44:23.666460 16252 consensus_queue.cc:1035] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:44:23.674396 16260 mvcc.cc:205] Tried to move back new op lower bound from 7298611051165519872 to 7298611051055812608. Current Snapshot: MvccSnapshot[applied={T|T < 7298611051165519872}]
I20260619 16:44:23.676952 16262 mvcc.cc:205] Tried to move back new op lower bound from 7298611051165519872 to 7298611051055812608. Current Snapshot: MvccSnapshot[applied={T|T < 7298611051165519872}]
I20260619 16:44:23.679754 16261 mvcc.cc:205] Tried to move back new op lower bound from 7298611051165519872 to 7298611051055812608. Current Snapshot: MvccSnapshot[applied={T|T < 7298611051165519872}]
I20260619 16:44:23.688232 15770 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:44:23.688457 15770 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:44:23.691720 15770 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad: Sending DeleteTablet for 3 replicas of tablet c0ec67fc1761441196d3d861225d009a
I20260619 16:44:23.692673 16163 tablet_service.cc:1514] Processing DeleteTablet for tablet c0ec67fc1761441196d3d861225d009a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43390
I20260619 16:44:23.692667 15895 tablet_service.cc:1514] Processing DeleteTablet for tablet c0ec67fc1761441196d3d861225d009a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60526
I20260619 16:44:23.693416 16269 tablet_replica.cc:330] T c0ec67fc1761441196d3d861225d009a P 18ec02b6990242519665eb24c9fa5f14: stopping tablet replica
I20260619 16:44:23.693717 16029 tablet_service.cc:1514] Processing DeleteTablet for tablet c0ec67fc1761441196d3d861225d009a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35770
I20260619 16:44:23.693954 16269 raft_consensus.cc:2229] T c0ec67fc1761441196d3d861225d009a P 18ec02b6990242519665eb24c9fa5f14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:44:23.694351 16269 raft_consensus.cc:2258] T c0ec67fc1761441196d3d861225d009a P 18ec02b6990242519665eb24c9fa5f14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:44:23.694661 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 15832
I20260619 16:44:23.696796 16271 tablet_replica.cc:330] T c0ec67fc1761441196d3d861225d009a P 53176a8d1367494f9eeb38acefecfa0c: stopping tablet replica
I20260619 16:44:23.696870 16269 ts_tablet_manager.cc:1905] T c0ec67fc1761441196d3d861225d009a P 18ec02b6990242519665eb24c9fa5f14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:44:23.697171 16271 raft_consensus.cc:2229] T c0ec67fc1761441196d3d861225d009a P 53176a8d1367494f9eeb38acefecfa0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:44:23.697561 16271 raft_consensus.cc:2258] T c0ec67fc1761441196d3d861225d009a P 53176a8d1367494f9eeb38acefecfa0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:44:23.700114 16269 ts_tablet_manager.cc:1918] T c0ec67fc1761441196d3d861225d009a P 18ec02b6990242519665eb24c9fa5f14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:44:23.700227 16269 log.cc:1196] T c0ec67fc1761441196d3d861225d009a P 18ec02b6990242519665eb24c9fa5f14: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/wal/wals/c0ec67fc1761441196d3d861225d009a
I20260619 16:44:23.700568 16269 ts_tablet_manager.cc:1939] T c0ec67fc1761441196d3d861225d009a P 18ec02b6990242519665eb24c9fa5f14: Deleting consensus metadata
I20260619 16:44:23.702059 15756 catalog_manager.cc:4841] TS 18ec02b6990242519665eb24c9fa5f14 (127.0.98.3:43281): tablet c0ec67fc1761441196d3d861225d009a (table test-table [id=1238014d302f40b2aa84b2e4507cbfdb]) successfully deleted
W20260619 16:44:23.703258 15754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44505 (error 108)
I20260619 16:44:23.704299 16271 ts_tablet_manager.cc:1905] T c0ec67fc1761441196d3d861225d009a P 53176a8d1367494f9eeb38acefecfa0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:44:23.704425 16240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:23.704118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44505 (local address 127.0.0.1:60550)
0619 16:44:23.704266 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:23.704343 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 16:44:23.704583 15754 catalog_manager.cc:4584] TS 869d5f0889b5494d8fb502c4b810df76 (127.0.98.1:44505): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c0ec67fc1761441196d3d861225d009a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44505: connect: Connection refused (error 111)
I20260619 16:44:23.706681 16271 ts_tablet_manager.cc:1918] T c0ec67fc1761441196d3d861225d009a P 53176a8d1367494f9eeb38acefecfa0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:44:23.706780 16271 log.cc:1196] T c0ec67fc1761441196d3d861225d009a P 53176a8d1367494f9eeb38acefecfa0c: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/wal/wals/c0ec67fc1761441196d3d861225d009a
I20260619 16:44:23.707116 16271 ts_tablet_manager.cc:1939] T c0ec67fc1761441196d3d861225d009a P 53176a8d1367494f9eeb38acefecfa0c: Deleting consensus metadata
I20260619 16:44:23.708561 15757 catalog_manager.cc:4841] TS 53176a8d1367494f9eeb38acefecfa0c (127.0.98.2:32803): tablet c0ec67fc1761441196d3d861225d009a (table test-table [id=1238014d302f40b2aa84b2e4507cbfdb]) successfully deleted
I20260619 16:44:23.708626 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 15966
I20260619 16:44:23.714836 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 16100
I20260619 16:44:23.721112 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 15738
I20260619 16:44:43.728111 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46833
--webserver_interface=127.0.98.62
--webserver_port=43313
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:44:43.866887 16273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:43.867280 16273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:43.867327 16273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:43.872161 16273 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 16:44:43.872262 16273 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 16:44:43.872280 16273 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 16:44:43.872295 16273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:44:43.872318 16273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:43.872337 16273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:43.872354 16273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:44:43.872371 16273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:44:43.879187 16273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:44:43.880815 16273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:43.882187 16273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:43.898918 16279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:43.899087 16273 server_base.cc:1034] running on GCE node
W20260619 16:44:43.899108 16280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:43.898928 16282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:43.900283 16273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:43.901906 16273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:43.903151 16273 hybrid_clock.cc:648] HybridClock initialized: now 1781887483903119 us; error 59 us; skew 500 ppm
Jun 19 16:44:43 dist-test-slave-03cd krb5kdc[15725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887483, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:43.906973 16273 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:44:43.908440 16273 webserver.cc:460] Webserver started at http://127.0.98.62:43313/ using document root <none> and password file <none>
I20260619 16:44:43.909199 16273 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:43.909294 16273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:43.914877 16273 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:44:43.918325 16289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:43.920435 16273 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 16:44:43.920666 16273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "e90ee4f9a07e4d23b0b9a9906cb04aad"
format_stamp: "Formatted at 2026-06-19 16:44:22 on dist-test-slave-03cd"
server_key: "ca9c8c19d293685c606c11c15c9f9c1b"
server_key_iv: "587c2dc0fc606558e52d19f272a98a4f"
server_key_version: "encryptionkey@0"
I20260619 16:44:43.921527 16273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:43.949180 16273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:43.953200 16273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:43.953557 16273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:43.963383 16273 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46833
I20260619 16:44:43.963837 16341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46833 every 8 connection(s)
I20260619 16:44:43.964947 16273 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:44:43.968364 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 16273
I20260619 16:44:43.969033 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44505
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46833
--webserver_port=34003
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:44:43.971184 16342 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:44:43.973470 16342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad: Bootstrap starting.
I20260619 16:44:43.993459 16342 log.cc:824] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad: Log is configured to *not* fsync() on all Append() calls
I20260619 16:44:44.007274 16342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:44:44.008104 16342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad: Bootstrap complete.
I20260619 16:44:44.015026 16342 raft_consensus.cc:348] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90ee4f9a07e4d23b0b9a9906cb04aad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46833 } }
I20260619 16:44:44.015755 16342 raft_consensus.cc:729] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e90ee4f9a07e4d23b0b9a9906cb04aad, State: Initialized, Role: FOLLOWER
I20260619 16:44:44.016458 16342 consensus_queue.cc:260] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90ee4f9a07e4d23b0b9a9906cb04aad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46833 } }
I20260619 16:44:44.016655 16342 raft_consensus.cc:388] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:44:44.016748 16342 raft_consensus.cc:482] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:44:44.016857 16342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:44:44.020407 16342 raft_consensus.cc:504] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90ee4f9a07e4d23b0b9a9906cb04aad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46833 } }
I20260619 16:44:44.020938 16342 leader_election.cc:302] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e90ee4f9a07e4d23b0b9a9906cb04aad; no voters:
I20260619 16:44:44.021430 16342 leader_election.cc:288] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 16:44:44.021731 16346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:44:44.022210 16346 raft_consensus.cc:686] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [term 2 LEADER]: Becoming Leader. State: Replica: e90ee4f9a07e4d23b0b9a9906cb04aad, State: Running, Role: LEADER
I20260619 16:44:44.022675 16346 consensus_queue.cc:237] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90ee4f9a07e4d23b0b9a9906cb04aad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46833 } }
I20260619 16:44:44.023131 16342 sys_catalog.cc:564] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:44:44.024441 16347 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e90ee4f9a07e4d23b0b9a9906cb04aad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90ee4f9a07e4d23b0b9a9906cb04aad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46833 } } }
I20260619 16:44:44.024631 16347 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [sys.catalog]: This master's current role is: LEADER
I20260619 16:44:44.025452 16348 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [sys.catalog]: SysCatalogTable state changed. Reason: New leader e90ee4f9a07e4d23b0b9a9906cb04aad. Latest consensus state: current_term: 2 leader_uuid: "e90ee4f9a07e4d23b0b9a9906cb04aad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90ee4f9a07e4d23b0b9a9906cb04aad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46833 } } }
I20260619 16:44:44.025583 16348 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [sys.catalog]: This master's current role is: LEADER
I20260619 16:44:44.025660 16355 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:44:44.031854 16355 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:44:44.033043 16355 catalog_manager.cc:1259] Loaded cluster ID: 2ad0be05d5354c1f96ef6fd1f980397a
I20260619 16:44:44.033145 16355 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:44:44.036249 16355 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:44:44.037271 16355 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad: Loaded TSK: 0
I20260619 16:44:44.058624 16355 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad: Generated new TSK 1
I20260619 16:44:44.058979 16355 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:44:44.127064 16344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:44.127522 16344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:44.127638 16344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:44.133085 16344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:44.133201 16344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:44.133281 16344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:44:44.140030 16344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34003
--tserver_master_addrs=127.0.98.62:46833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:44:44.141583 16344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:44.143019 16344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:44.155135 16371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:44.155150 16373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:44.155362 16370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:44.155653 16344 server_base.cc:1034] running on GCE node
I20260619 16:44:44.156272 16344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:44.157210 16344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:44.158492 16344 hybrid_clock.cc:648] HybridClock initialized: now 1781887484158439 us; error 75 us; skew 500 ppm
Jun 19 16:44:44 dist-test-slave-03cd krb5kdc[15725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:44.162503 16344 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:44:44.163887 16344 webserver.cc:460] Webserver started at http://127.0.98.1:34003/ using document root <none> and password file <none>
I20260619 16:44:44.164615 16344 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:44.164743 16344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:44.169747 16344 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 16:44:44.172730 16380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:44.174516 16344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:44:44.174719 16344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "869d5f0889b5494d8fb502c4b810df76"
format_stamp: "Formatted at 2026-06-19 16:44:22 on dist-test-slave-03cd"
server_key: "57fb264417f98af538e63f3865d456eb"
server_key_iv: "bcdb807774b4ffd66a31df8714c9f22b"
server_key_version: "encryptionkey@0"
I20260619 16:44:44.175439 16344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:44.208215 16344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:44.212648 16344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:44.213104 16344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:44.213944 16344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:44:44.215493 16387 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 16:44:44.220289 16344 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 16:44:44.220436 16344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260619 16:44:44.220595 16344 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 16:44:44.223008 16344 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 16:44:44.223099 16344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260619 16:44:44.223506 16387 tablet_bootstrap.cc:492] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76: Bootstrap starting.
I20260619 16:44:44.236363 16344 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44505
I20260619 16:44:44.236471 16494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44505 every 8 connection(s)
I20260619 16:44:44.237628 16344 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:44:44.239387 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 16344
I20260619 16:44:44.239984 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:32803
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46833
--webserver_port=44013
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:44:44.245916 16387 log.cc:824] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76: Log is configured to *not* fsync() on all Append() calls
Jun 19 16:44:44 dist-test-slave-03cd krb5kdc[15725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:44:44.256669 16387 tablet_bootstrap.cc:492] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:44:44.257185 16387 tablet_bootstrap.cc:492] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76: Bootstrap complete.
I20260619 16:44:44.258222 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:44.240271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.1:55079)
0619 16:44:44.241746 (+ 1475us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:44.241818 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:44.243179 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:44.244622 (+ 1443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:44.244631 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:44.245206 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:44.245784 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:44.245801 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.247590 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.247593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:44.248631 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:44.248642 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.249446 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.249450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:44.249687 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:44.250435 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:44.250453 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:44.252206 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:44.254690 (+ 2484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:44.254697 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:44.254713 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:44.255003 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:44.255463 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:44.255466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:44.255468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:44.255616 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:44.256326 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:44.256333 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:44.256627 (+ 294us) client_negotiation.cc:769] Sending connection context
0619 16:44:44.256827 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 16:44:44.257206 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1154,"thread_start_us":151,"threads_started":1}
I20260619 16:44:44.258684 16387 ts_tablet_manager.cc:1397] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76: Time spent bootstrapping tablet: real 0.035s user 0.028s sys 0.003s
I20260619 16:44:44.260166 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:44.240257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55079 (local address 127.0.98.62:46833)
0619 16:44:44.242086 (+ 1829us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:44.242110 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:44.242176 (+ 66us) server_negotiation.cc:407] Connection header received
0619 16:44:44.243803 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:44.243822 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:44.244188 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:44.244559 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:44.245980 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.247406 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.248767 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.249271 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.249467 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:44.252471 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:44.252478 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:44.252492 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:44.252526 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:44.254534 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:44.255145 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:44.255151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:44.255155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:44.255318 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:44.255758 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:44.255762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:44.255764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:44.256196 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:44.256349 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:44.258810 (+ 2461us) server_negotiation.cc:299] Negotiation successful
0619 16:44:44.259358 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1608,"thread_start_us":132,"threads_started":1}
I20260619 16:44:44.262596 16495 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46833
I20260619 16:44:44.262194 16387 raft_consensus.cc:348] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 } } peers { permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 } } peers { permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } }
I20260619 16:44:44.262743 16387 raft_consensus.cc:729] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869d5f0889b5494d8fb502c4b810df76, State: Initialized, Role: FOLLOWER
I20260619 16:44:44.262916 16495 heartbeater.cc:464] Registering TS with master...
I20260619 16:44:44.263377 16387 consensus_queue.cc:260] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 } } peers { permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 } } peers { permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } }
I20260619 16:44:44.263639 16495 heartbeater.cc:511] Master 127.0.98.62:46833 requested a full tablet report, sending...
I20260619 16:44:44.264557 16387 ts_tablet_manager.cc:1428] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20260619 16:44:44.266961 16305 ts_manager.cc:194] Registered new tserver with Master: 869d5f0889b5494d8fb502c4b810df76 (127.0.98.1:44505)
I20260619 16:44:44.271235 16305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55079
I20260619 16:44:44.274564 16495 heartbeater.cc:503] Master 127.0.98.62:46833 was elected leader, sending a full tablet report...
Jun 19 16:44:44 dist-test-slave-03cd krb5kdc[15725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887483, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:44:44.284116 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:44.270927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49424 (local address 127.0.98.1:44505)
0619 16:44:44.271475 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:44.271488 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:44.271994 (+ 506us) server_negotiation.cc:407] Connection header received
0619 16:44:44.272324 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:44.272329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:44.272601 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:44.272758 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:44.273525 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.274307 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.275518 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.276140 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.276300 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:44.279369 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:44.279381 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:44.279397 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:44.279428 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:44.281610 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:44.282338 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:44.282344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:44.282345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:44.282450 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:44.282784 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:44.282786 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:44.282788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:44.283099 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:44.283230 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:44.283785 (+ 555us) server_negotiation.cc:299] Negotiation successful
0619 16:44:44.283961 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":132,"threads_started":1}
I20260619 16:44:44.284116 16502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:44.270843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44505 (local address 127.0.0.1:49424)
0619 16:44:44.271827 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:44.271873 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:44.272133 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:44.272780 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:44.272787 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:44.273255 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:44.273394 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:44.273401 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.274440 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.274444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:44.275339 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:44.275350 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.276300 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.276325 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:44.276443 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:44.277213 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:44.277226 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:44.279140 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:44.281780 (+ 2640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:44.281800 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:44.281807 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:44.282195 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:44.282561 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:44.282564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:44.282568 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:44.282676 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:44.283242 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:44.283252 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:44.283445 (+ 193us) client_negotiation.cc:769] Sending connection context
0619 16:44:44.283626 (+ 181us) client_negotiation.cc:240] Negotiation successful
0619 16:44:44.283909 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":169,"threads_started":1}
I20260619 16:44:44.285701 16429 tablet_service.cc:1514] Processing DeleteTablet for tablet c0ec67fc1761441196d3d861225d009a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49424
I20260619 16:44:44.286924 16504 tablet_replica.cc:330] stopping tablet replica
I20260619 16:44:44.287354 16504 raft_consensus.cc:2229] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:44:44.287758 16504 raft_consensus.cc:2258] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:44:44.289147 16504 ts_tablet_manager.cc:1905] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:44:44.293906 16504 ts_tablet_manager.cc:1918] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:44:44.294059 16504 log.cc:1196] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/wal/wals/c0ec67fc1761441196d3d861225d009a
I20260619 16:44:44.294369 16504 ts_tablet_manager.cc:1939] T c0ec67fc1761441196d3d861225d009a P 869d5f0889b5494d8fb502c4b810df76: Deleting consensus metadata
I20260619 16:44:44.296795 16290 catalog_manager.cc:4841] TS 869d5f0889b5494d8fb502c4b810df76 (127.0.98.1:44505): tablet c0ec67fc1761441196d3d861225d009a (table test-table [id=1238014d302f40b2aa84b2e4507cbfdb]) successfully deleted
W20260619 16:44:44.379069 16499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:44.379398 16499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:44.379498 16499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:44.384536 16499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:44.384630 16499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:44.384727 16499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:44:44.390878 16499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:32803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44013
--tserver_master_addrs=127.0.98.62:46833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:44:44.392295 16499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:44.393428 16499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:44.404438 16510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:44.404634 16499 server_base.cc:1034] running on GCE node
W20260619 16:44:44.404448 16512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:44.404439 16509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:44.405292 16499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:44.405965 16499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:44.407176 16499 hybrid_clock.cc:648] HybridClock initialized: now 1781887484407140 us; error 51 us; skew 500 ppm
Jun 19 16:44:44 dist-test-slave-03cd krb5kdc[15725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:44.410429 16499 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:44:44.411407 16499 webserver.cc:460] Webserver started at http://127.0.98.2:44013/ using document root <none> and password file <none>
I20260619 16:44:44.412128 16499 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:44.412210 16499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:44.416451 16499 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:44:44.418720 16519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:44.420109 16499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:44:44.420238 16499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "53176a8d1367494f9eeb38acefecfa0c"
format_stamp: "Formatted at 2026-06-19 16:44:23 on dist-test-slave-03cd"
server_key: "18caedd2a847cdea4ba46431788b9649"
server_key_iv: "a6743a74054908d353ad32c88c9f0e31"
server_key_version: "encryptionkey@0"
I20260619 16:44:44.420806 16499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:44.441937 16499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:44.445303 16499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:44.445648 16499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:44.446362 16499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:44:44.447304 16499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:44:44.447355 16499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:44.447405 16499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:44:44.447465 16499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:44.457576 16499 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32803
I20260619 16:44:44.457620 16632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32803 every 8 connection(s)
I20260619 16:44:44.458657 16499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:44:44.459589 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 16499
I20260619 16:44:44.460135 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43281
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46833
--webserver_port=45957
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:44:44 dist-test-slave-03cd krb5kdc[15725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:44:44.474876 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:44.460936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55589 (local address 127.0.98.62:46833)
0619 16:44:44.461093 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:44.461097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:44.462334 (+ 1237us) server_negotiation.cc:407] Connection header received
0619 16:44:44.464035 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:44.464038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:44.464101 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:44.464204 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:44.465457 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.466017 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.467196 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.467451 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.467568 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:44.470546 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:44.470549 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:44.470551 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:44.470580 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:44.472274 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:44.472983 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:44.472986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:44.472988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:44.473072 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:44.473526 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:44.473529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:44.473531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:44.473733 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:44.473824 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:44.474528 (+ 704us) server_negotiation.cc:299] Negotiation successful
0619 16:44:44.474714 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:44:44.475967 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:44.461480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.2:55589)
0619 16:44:44.462143 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:44.462210 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:44.463796 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:44.464384 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:44.464391 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:44.464772 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:44.465289 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:44.465303 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.466151 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.466161 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:44.467066 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:44.467076 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.467617 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.467621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:44.467850 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:44.468526 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:44.468542 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:44.470284 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:44.472422 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:44.472430 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:44.472451 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:44.472852 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:44.473197 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:44.473200 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:44.473202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:44.473401 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:44.473841 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:44.473848 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:44.474218 (+ 370us) client_negotiation.cc:769] Sending connection context
0619 16:44:44.474446 (+ 228us) client_negotiation.cc:240] Negotiation successful
0619 16:44:44.474839 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":181,"threads_started":1}
I20260619 16:44:44.477373 16633 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46833
I20260619 16:44:44.477670 16633 heartbeater.cc:464] Registering TS with master...
I20260619 16:44:44.478247 16633 heartbeater.cc:511] Master 127.0.98.62:46833 requested a full tablet report, sending...
I20260619 16:44:44.479566 16305 ts_manager.cc:194] Registered new tserver with Master: 53176a8d1367494f9eeb38acefecfa0c (127.0.98.2:32803)
I20260619 16:44:44.480440 16305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55589
W20260619 16:44:44.580456 16635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:44.580767 16635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:44.580814 16635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:44.584968 16635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:44.585044 16635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:44.585107 16635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:44:44.590624 16635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45957
--tserver_master_addrs=127.0.98.62:46833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:44:44.591774 16635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:44.592772 16635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:44.603021 16644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:44.603085 16642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:44.603021 16641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:44.603327 16635 server_base.cc:1034] running on GCE node
I20260619 16:44:44.603992 16635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:44.604751 16635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:44.606001 16635 hybrid_clock.cc:648] HybridClock initialized: now 1781887484605988 us; error 34 us; skew 500 ppm
Jun 19 16:44:44 dist-test-slave-03cd krb5kdc[15725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:44.609462 16635 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:44:44.610487 16635 webserver.cc:460] Webserver started at http://127.0.98.3:45957/ using document root <none> and password file <none>
I20260619 16:44:44.611112 16635 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:44.611235 16635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:44.615248 16635 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:44:44.617784 16651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:44.619237 16635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:44:44.619400 16635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "18ec02b6990242519665eb24c9fa5f14"
format_stamp: "Formatted at 2026-06-19 16:44:23 on dist-test-slave-03cd"
server_key: "0907382f51814a0c728ce713b280cb08"
server_key_iv: "ff78012f373e85958ca966a514180804"
server_key_version: "encryptionkey@0"
I20260619 16:44:44.620116 16635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:44.639446 16635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:44.642961 16635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:44.643210 16635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:44.643944 16635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:44:44.644804 16635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:44:44.644886 16635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:44.644958 16635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:44:44.645001 16635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:44.654310 16635 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43281
I20260619 16:44:44.654335 16764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43281 every 8 connection(s)
I20260619 16:44:44.655318 16635 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:44:44.658159 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 16635
Jun 19 16:44:44 dist-test-slave-03cd krb5kdc[15725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:44:44.669387 16769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:44.658868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52688 (local address 127.0.98.62:46833)
0619 16:44:44.659103 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:44.659107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:44.660339 (+ 1232us) server_negotiation.cc:407] Connection header received
0619 16:44:44.660562 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:44.660565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:44.660606 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:44.660691 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:44.661562 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.662327 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.663551 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.663865 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.663952 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:44.665645 (+ 1693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:44.665649 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:44.665651 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:44.665681 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:44.667391 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:44.667983 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:44.667986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:44.667987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:44.668069 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:44.668375 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:44.668379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:44.668382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:44.668634 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:44.668791 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:44.669052 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 16:44:44.669191 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":56,"threads_started":1}
I20260619 16:44:44.672034 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:44.657142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52513 (local address 127.0.98.62:46833)
0619 16:44:44.657280 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:44.657283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:44.658120 (+ 837us) server_negotiation.cc:407] Connection header received
0619 16:44:44.660090 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:44.660094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:44.660151 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:44.660223 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:44.661885 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.662553 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.664132 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.664329 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.664443 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:44.667325 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:44.667328 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:44.667330 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:44.667366 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:44.669283 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:44.670004 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:44.670007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:44.670009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:44.670089 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:44.670647 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:44.670650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:44.670652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:44.670895 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:44.670992 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:44.671702 (+ 710us) server_negotiation.cc:299] Negotiation successful
0619 16:44:44.671849 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55,"spinlock_wait_cycles":896}
I20260619 16:44:44.672950 16767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:44.657525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.3:52513)
0619 16:44:44.657975 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:44.658014 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:44.659833 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:44.660469 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:44.660480 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:44.661021 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:44.661717 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:44.661733 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.662691 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.662696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:44.664014 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:44.664025 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.664500 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.664506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:44.664692 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:44.665237 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:44.665253 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:44.667122 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:44.669432 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:44.669438 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:44.669454 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:44.669892 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:44.670219 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:44.670226 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:44.670230 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:44.670527 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:44.671027 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:44.671035 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:44.671385 (+ 350us) client_negotiation.cc:769] Sending connection context
0619 16:44:44.671628 (+ 243us) client_negotiation.cc:240] Negotiation successful
0619 16:44:44.672040 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":130,"threads_started":1}
I20260619 16:44:44.674039 16765 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46833
I20260619 16:44:44.674307 16765 heartbeater.cc:464] Registering TS with master...
I20260619 16:44:44.674873 16765 heartbeater.cc:511] Master 127.0.98.62:46833 requested a full tablet report, sending...
I20260619 16:44:44.676055 16305 ts_manager.cc:194] Registered new tserver with Master: 18ec02b6990242519665eb24c9fa5f14 (127.0.98.3:43281)
I20260619 16:44:44.676648 16305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52513
I20260619 16:44:44.684273 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:44:44.688925 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:44.685069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52702 (local address 127.0.98.62:46833)
0619 16:44:44.685252 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:44.685258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:44.685439 (+ 181us) server_negotiation.cc:407] Connection header received
0619 16:44:44.685665 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:44.685667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:44.685783 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:44.685904 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:44:44.686152 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.686680 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.687563 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.687865 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.687979 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:44.688317 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:44:44.688475 (+ 158us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 16:44:44.688761 (+ 286us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52702: token expired
Metrics: {"server-negotiator.queue_time_us":86}
W20260619 16:44:44.688999 16497 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52702: token expired
W20260619 16:44:44.689366 16234 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:46833)
I20260619 16:44:44.697136 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:44.689745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52704 (local address 127.0.98.62:46833)
0619 16:44:44.689861 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:44.689869 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:44.689881 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:44:44.689956 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:44.689958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:44.689995 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:44.690062 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:44.690685 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.691176 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.692032 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.692248 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.692328 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:44.693778 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:44.693781 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:44.693783 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:44.693808 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:44.695387 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:44.695925 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:44.695928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:44.695929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:44.696002 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:44.696276 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:44.696281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:44.696283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:44.696474 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:44.696613 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:44.696850 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 16:44:44.696963 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 16:44:44.699872 16305 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:44:44.703454 16305 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:44:44.711900 16429 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7964a1aaf34e2086ccff174aa0f865 (DEFAULT_TABLE table=test-table [id=ed1df4bdbd0e42708bbd07710ef22e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:44:44.712610 16429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b7964a1aaf34e2086ccff174aa0f865. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:44.715397 16387 tablet_bootstrap.cc:492] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76: Bootstrap starting.
I20260619 16:44:44.716382 16387 tablet_bootstrap.cc:654] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:44.717120 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:44.710996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58224 (local address 127.0.98.3:43281)
0619 16:44:44.711932 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:44.711940 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:44.711959 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:44:44.712020 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:44.712024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:44.712259 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:44.712862 (+ 603us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:44.713063 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.713803 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.715808 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.716532 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.716641 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:44.716794 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:44.716887 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 16:44:44.716982 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":831,"thread_start_us":153,"threads_started":1}
I20260619 16:44:44.717108 16502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:44.710808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43281 (local address 127.0.0.1:58224)
0619 16:44:44.710962 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:44.710980 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:44.711060 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:44.712425 (+ 1365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:44.712428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:44.712446 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:44.712584 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:44.712591 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.713940 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.713946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:44.715638 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:44.715646 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.716642 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.716645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:44.716740 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:44.716786 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 16:44:44.716859 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 16:44:44.716965 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":273}
I20260619 16:44:44.717331 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:44.710985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32803 (local address 127.0.0.1:37906)
0619 16:44:44.711493 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:44.711508 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:44.711641 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:44.712406 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:44.712409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:44.712427 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:44.712705 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:44.712711 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.714042 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.714045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:44.715372 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:44.715383 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.716899 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.716903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:44.717004 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:44.717039 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 16:44:44.717096 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 16:44:44.717172 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":242,"thread_start_us":136,"threads_started":1}
I20260619 16:44:44.717989 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:44.711440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37906 (local address 127.0.98.2:32803)
0619 16:44:44.711888 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:44.711901 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:44.711927 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:44:44.712010 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:44.712015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:44.712229 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:44.712581 (+ 352us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:44.712814 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.713879 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.715637 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.716701 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.717516 (+ 815us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:44.717664 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:44.717714 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 16:44:44.717822 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":212,"threads_started":1}
I20260619 16:44:44.718259 16387 tablet_bootstrap.cc:492] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76: No bootstrap required, opened a new log
I20260619 16:44:44.718334 16387 ts_tablet_manager.cc:1397] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260619 16:44:44.718900 16387 raft_consensus.cc:348] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 } } peers { permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 } }
I20260619 16:44:44.719050 16387 raft_consensus.cc:374] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:44.719091 16387 raft_consensus.cc:729] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869d5f0889b5494d8fb502c4b810df76, State: Initialized, Role: FOLLOWER
I20260619 16:44:44.721268 16699 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7964a1aaf34e2086ccff174aa0f865 (DEFAULT_TABLE table=test-table [id=ed1df4bdbd0e42708bbd07710ef22e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:44:44.721267 16567 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7964a1aaf34e2086ccff174aa0f865 (DEFAULT_TABLE table=test-table [id=ed1df4bdbd0e42708bbd07710ef22e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:44:44.722039 16387 consensus_queue.cc:260] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 } } peers { permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 } }
I20260619 16:44:44.722471 16699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b7964a1aaf34e2086ccff174aa0f865. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:44.722465 16567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b7964a1aaf34e2086ccff174aa0f865. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:44.722497 16387 ts_tablet_manager.cc:1428] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260619 16:44:44.732278 16778 tablet_bootstrap.cc:492] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c: Bootstrap starting.
I20260619 16:44:44.734570 16778 tablet_bootstrap.cc:654] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:44.735270 16778 log.cc:824] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:44:44.735898 16779 tablet_bootstrap.cc:492] T 3b7964a1aaf34e2086ccff174aa0f865 P 18ec02b6990242519665eb24c9fa5f14: Bootstrap starting.
I20260619 16:44:44.737305 16778 tablet_bootstrap.cc:492] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c: No bootstrap required, opened a new log
I20260619 16:44:44.737555 16778 ts_tablet_manager.cc:1397] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 16:44:44.737740 16779 tablet_bootstrap.cc:654] T 3b7964a1aaf34e2086ccff174aa0f865 P 18ec02b6990242519665eb24c9fa5f14: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:44.738729 16779 log.cc:824] T 3b7964a1aaf34e2086ccff174aa0f865 P 18ec02b6990242519665eb24c9fa5f14: Log is configured to *not* fsync() on all Append() calls
W20260619 16:44:44.739144 16496 tablet.cc:2369] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:44:44.740996 16779 tablet_bootstrap.cc:492] T 3b7964a1aaf34e2086ccff174aa0f865 P 18ec02b6990242519665eb24c9fa5f14: No bootstrap required, opened a new log
I20260619 16:44:44.741216 16779 ts_tablet_manager.cc:1397] T 3b7964a1aaf34e2086ccff174aa0f865 P 18ec02b6990242519665eb24c9fa5f14: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 16:44:44.741362 16778 raft_consensus.cc:348] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 } } peers { permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 } }
I20260619 16:44:44.741624 16778 raft_consensus.cc:374] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:44.741709 16778 raft_consensus.cc:729] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53176a8d1367494f9eeb38acefecfa0c, State: Initialized, Role: FOLLOWER
I20260619 16:44:44.742125 16778 consensus_queue.cc:260] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 } } peers { permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 } }
I20260619 16:44:44.742832 16633 heartbeater.cc:503] Master 127.0.98.62:46833 was elected leader, sending a full tablet report...
I20260619 16:44:44.743081 16778 ts_tablet_manager.cc:1428] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 16:44:44.744701 16779 raft_consensus.cc:348] T 3b7964a1aaf34e2086ccff174aa0f865 P 18ec02b6990242519665eb24c9fa5f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 } } peers { permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 } }
I20260619 16:44:44.745007 16779 raft_consensus.cc:374] T 3b7964a1aaf34e2086ccff174aa0f865 P 18ec02b6990242519665eb24c9fa5f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:44.745074 16779 raft_consensus.cc:729] T 3b7964a1aaf34e2086ccff174aa0f865 P 18ec02b6990242519665eb24c9fa5f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18ec02b6990242519665eb24c9fa5f14, State: Initialized, Role: FOLLOWER
I20260619 16:44:44.745457 16779 consensus_queue.cc:260] T 3b7964a1aaf34e2086ccff174aa0f865 P 18ec02b6990242519665eb24c9fa5f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 } } peers { permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 } }
I20260619 16:44:44.746121 16765 heartbeater.cc:503] Master 127.0.98.62:46833 was elected leader, sending a full tablet report...
I20260619 16:44:44.746430 16779 ts_tablet_manager.cc:1428] T 3b7964a1aaf34e2086ccff174aa0f865 P 18ec02b6990242519665eb24c9fa5f14: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 16:44:44.897884 16782 raft_consensus.cc:482] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:44:44.898095 16782 raft_consensus.cc:504] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 } } peers { permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 } }
I20260619 16:44:44.899322 16782 leader_election.cc:288] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18ec02b6990242519665eb24c9fa5f14 (127.0.98.3:43281), 869d5f0889b5494d8fb502c4b810df76 (127.0.98.1:44505)
I20260619 16:44:44.904752 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:44.899748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58007 (local address 127.0.98.1:44505)
0619 16:44:44.900202 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:44.900207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:44.900221 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:44:44.900287 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:44.900289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:44.900382 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:44.900499 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:44.900870 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.901445 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.903349 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.904060 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.904156 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:44.904319 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:44.904530 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 16:44:44.904612 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":94,"threads_started":1}
I20260619 16:44:44.904748 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:44.900031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53607 (local address 127.0.98.3:43281)
0619 16:44:44.900202 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:44.900207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:44.900221 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:44:44.900376 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:44.900379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:44.900429 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:44.900501 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:44.900938 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.901428 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.903309 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.903967 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.904087 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:44.904171 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:44.904530 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 16:44:44.904608 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260619 16:44:44.904781 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:44.899584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44505 (local address 127.0.98.2:58007)
0619 16:44:44.899927 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:44.899944 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:44.900046 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:44.900533 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:44.900539 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:44.900561 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:44.900713 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:44.900722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.901565 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.901572 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:44.903145 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:44.903159 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.904172 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.904175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:44.904317 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:44.904423 (+ 106us) client_negotiation.cc:769] Sending connection context
0619 16:44:44.904498 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 16:44:44.904617 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":102}
I20260619 16:44:44.904790 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:44.899622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43281 (local address 127.0.98.2:53607)
0619 16:44:44.900139 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:44.900154 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:44.900254 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:44.900615 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:44.900618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:44.900642 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:44.900768 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:44.900774 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.901542 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.901547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:44.903145 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:44.903159 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.904155 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.904160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:44.904312 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:44.904393 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 16:44:44.904488 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 16:44:44.904617 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":236,"thread_start_us":167,"threads_started":1}
I20260619 16:44:44.905408 16449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7964a1aaf34e2086ccff174aa0f865" candidate_uuid: "53176a8d1367494f9eeb38acefecfa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869d5f0889b5494d8fb502c4b810df76" is_pre_election: true
I20260619 16:44:44.905613 16719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7964a1aaf34e2086ccff174aa0f865" candidate_uuid: "53176a8d1367494f9eeb38acefecfa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18ec02b6990242519665eb24c9fa5f14" is_pre_election: true
I20260619 16:44:44.905721 16449 raft_consensus.cc:2454] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53176a8d1367494f9eeb38acefecfa0c in term 0.
I20260619 16:44:44.905910 16719 raft_consensus.cc:2454] T 3b7964a1aaf34e2086ccff174aa0f865 P 18ec02b6990242519665eb24c9fa5f14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53176a8d1367494f9eeb38acefecfa0c in term 0.
W20260619 16:44:44.906219 16766 tablet.cc:2369] T 3b7964a1aaf34e2086ccff174aa0f865 P 18ec02b6990242519665eb24c9fa5f14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:44:44.906227 16520 leader_election.cc:302] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53176a8d1367494f9eeb38acefecfa0c, 869d5f0889b5494d8fb502c4b810df76; no voters:
I20260619 16:44:44.906505 16782 raft_consensus.cc:2781] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:44:44.906576 16782 raft_consensus.cc:482] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:44:44.906617 16782 raft_consensus.cc:3037] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:44.907691 16782 raft_consensus.cc:504] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 } } peers { permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 } }
I20260619 16:44:44.908082 16782 leader_election.cc:288] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c [CANDIDATE]: Term 1 election: Requested vote from peers 18ec02b6990242519665eb24c9fa5f14 (127.0.98.3:43281), 869d5f0889b5494d8fb502c4b810df76 (127.0.98.1:44505)
I20260619 16:44:44.908541 16449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7964a1aaf34e2086ccff174aa0f865" candidate_uuid: "53176a8d1367494f9eeb38acefecfa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869d5f0889b5494d8fb502c4b810df76"
I20260619 16:44:44.908541 16719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7964a1aaf34e2086ccff174aa0f865" candidate_uuid: "53176a8d1367494f9eeb38acefecfa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18ec02b6990242519665eb24c9fa5f14"
I20260619 16:44:44.908685 16719 raft_consensus.cc:3037] T 3b7964a1aaf34e2086ccff174aa0f865 P 18ec02b6990242519665eb24c9fa5f14 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:44.908684 16449 raft_consensus.cc:3037] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:44.909989 16449 raft_consensus.cc:2454] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53176a8d1367494f9eeb38acefecfa0c in term 1.
I20260619 16:44:44.909989 16719 raft_consensus.cc:2454] T 3b7964a1aaf34e2086ccff174aa0f865 P 18ec02b6990242519665eb24c9fa5f14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53176a8d1367494f9eeb38acefecfa0c in term 1.
I20260619 16:44:44.910388 16522 leader_election.cc:302] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18ec02b6990242519665eb24c9fa5f14, 53176a8d1367494f9eeb38acefecfa0c; no voters:
I20260619 16:44:44.910575 16782 raft_consensus.cc:2781] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:44:44.910820 16782 raft_consensus.cc:686] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c [term 1 LEADER]: Becoming Leader. State: Replica: 53176a8d1367494f9eeb38acefecfa0c, State: Running, Role: LEADER
I20260619 16:44:44.911224 16782 consensus_queue.cc:237] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 } } peers { permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 } }
I20260619 16:44:44.915252 16305 catalog_manager.cc:5495] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 53176a8d1367494f9eeb38acefecfa0c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "53176a8d1367494f9eeb38acefecfa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 } health_report { overall_health: UNKNOWN } } }
I20260619 16:44:44.955060 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:44.951451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37908 (local address 127.0.98.2:32803)
0619 16:44:44.951593 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:44.951596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:44.951609 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:44:44.951708 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:44.951710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:44.951766 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:44.951864 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:44:44.952161 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.952702 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.953531 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:44.953747 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:44.953860 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:44.954182 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:44:44.954623 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:44:44.954758 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:44.954831 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 16:44:44.954893 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 16:44:44.959954 16634 tablet.cc:2369] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:44:44.961778 16719 raft_consensus.cc:1261] T 3b7964a1aaf34e2086ccff174aa0f865 P 18ec02b6990242519665eb24c9fa5f14 [term 1 FOLLOWER]: Refusing update from remote peer 53176a8d1367494f9eeb38acefecfa0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:44:44.961776 16449 raft_consensus.cc:1261] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [term 1 FOLLOWER]: Refusing update from remote peer 53176a8d1367494f9eeb38acefecfa0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:44:44.962523 16782 consensus_queue.cc:1035] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:44:44.962803 16786 consensus_queue.cc:1035] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:44:44.968650 16789 mvcc.cc:205] Tried to move back new op lower bound from 7298611138393731072 to 7298611138200428544. Current Snapshot: MvccSnapshot[applied={T|T < 7298611138393731072}]
I20260619 16:44:44.970356 16790 mvcc.cc:205] Tried to move back new op lower bound from 7298611138393731072 to 7298611138200428544. Current Snapshot: MvccSnapshot[applied={T|T < 7298611138393731072}]
I20260619 16:44:44.982558 16302 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:44:44.982771 16302 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:44:44.985253 16302 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad: Sending DeleteTablet for 3 replicas of tablet 3b7964a1aaf34e2086ccff174aa0f865
I20260619 16:44:44.986095 16699 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7964a1aaf34e2086ccff174aa0f865 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58224
I20260619 16:44:44.986256 16429 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7964a1aaf34e2086ccff174aa0f865 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49424
I20260619 16:44:44.986688 16798 tablet_replica.cc:330] T 3b7964a1aaf34e2086ccff174aa0f865 P 18ec02b6990242519665eb24c9fa5f14: stopping tablet replica
I20260619 16:44:44.986707 16567 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7964a1aaf34e2086ccff174aa0f865 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37906
I20260619 16:44:44.987069 16798 raft_consensus.cc:2229] T 3b7964a1aaf34e2086ccff174aa0f865 P 18ec02b6990242519665eb24c9fa5f14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:44:44.987218 16800 tablet_replica.cc:330] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c: stopping tablet replica
I20260619 16:44:44.987363 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 16344
I20260619 16:44:44.987445 16798 raft_consensus.cc:2258] T 3b7964a1aaf34e2086ccff174aa0f865 P 18ec02b6990242519665eb24c9fa5f14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:44:44.987639 16800 raft_consensus.cc:2229] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:44:44.988090 16800 raft_consensus.cc:2258] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:44:44.989544 16800 ts_tablet_manager.cc:1905] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:44:44.992540 16800 ts_tablet_manager.cc:1918] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:44:44.992654 16800 log.cc:1196] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/wal/wals/3b7964a1aaf34e2086ccff174aa0f865
I20260619 16:44:44.992976 16800 ts_tablet_manager.cc:1939] T 3b7964a1aaf34e2086ccff174aa0f865 P 53176a8d1367494f9eeb38acefecfa0c: Deleting consensus metadata
I20260619 16:44:44.993454 16798 ts_tablet_manager.cc:1905] T 3b7964a1aaf34e2086ccff174aa0f865 P 18ec02b6990242519665eb24c9fa5f14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:44:44.994300 16293 catalog_manager.cc:4841] TS 53176a8d1367494f9eeb38acefecfa0c (127.0.98.2:32803): tablet 3b7964a1aaf34e2086ccff174aa0f865 (table test-table [id=ed1df4bdbd0e42708bbd07710ef22e3a]) successfully deleted
W20260619 16:44:44.994879 16290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44505 (error 108)
W20260619 16:44:44.996248 16774 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:44.995945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44505 (local address 127.0.0.1:49426)
0619 16:44:44.996087 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:44.996158 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44505: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 16:44:44.996465 16290 catalog_manager.cc:4584] TS 869d5f0889b5494d8fb502c4b810df76 (127.0.98.1:44505): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3b7964a1aaf34e2086ccff174aa0f865: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44505: connect: Connection reset by peer (error 104)
I20260619 16:44:44.996886 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 16499
I20260619 16:44:44.998205 16798 ts_tablet_manager.cc:1918] T 3b7964a1aaf34e2086ccff174aa0f865 P 18ec02b6990242519665eb24c9fa5f14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:44:44.998343 16798 log.cc:1196] T 3b7964a1aaf34e2086ccff174aa0f865 P 18ec02b6990242519665eb24c9fa5f14: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/wal/wals/3b7964a1aaf34e2086ccff174aa0f865
I20260619 16:44:44.998669 16798 ts_tablet_manager.cc:1939] T 3b7964a1aaf34e2086ccff174aa0f865 P 18ec02b6990242519665eb24c9fa5f14: Deleting consensus metadata
I20260619 16:44:45.000192 16292 catalog_manager.cc:4841] TS 18ec02b6990242519665eb24c9fa5f14 (127.0.98.3:43281): tablet 3b7964a1aaf34e2086ccff174aa0f865 (table test-table [id=ed1df4bdbd0e42708bbd07710ef22e3a]) successfully deleted
I20260619 16:44:45.005792 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 16635
I20260619 16:44:45.012605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 16273
I20260619 16:45:05.020653 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46833
--webserver_interface=127.0.98.62
--webserver_port=43313
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:05.137100 16802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:05.137398 16802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:05.137440 16802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:05.141611 16802 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 16:45:05.141686 16802 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 16:45:05.141703 16802 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 16:45:05.141717 16802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:45:05.141737 16802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:05.141757 16802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:05.141772 16802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:45:05.141788 16802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:45:05.147552 16802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:05.148698 16802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:05.149683 16802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:45:05.159965 16802 server_base.cc:1034] running on GCE node
W20260619 16:45:05.160115 16810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:05.160130 16807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:05.160353 16808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:05.161244 16802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:05.162739 16802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:05.164037 16802 hybrid_clock.cc:648] HybridClock initialized: now 1781887505164009 us; error 49 us; skew 500 ppm
Jun 19 16:45:05 dist-test-slave-03cd krb5kdc[15725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:05.167644 16802 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:45:05.168926 16802 webserver.cc:460] Webserver started at http://127.0.98.62:43313/ using document root <none> and password file <none>
I20260619 16:45:05.169662 16802 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:05.169762 16802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:05.175014 16802 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260619 16:45:05.178200 16817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:05.180226 16802 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:45:05.180541 16802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "e90ee4f9a07e4d23b0b9a9906cb04aad"
format_stamp: "Formatted at 2026-06-19 16:44:22 on dist-test-slave-03cd"
server_key: "ca9c8c19d293685c606c11c15c9f9c1b"
server_key_iv: "587c2dc0fc606558e52d19f272a98a4f"
server_key_version: "encryptionkey@0"
I20260619 16:45:05.181414 16802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:05.218526 16802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:05.223728 16802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:05.224025 16802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:05.234200 16802 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46833
I20260619 16:45:05.234592 16869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46833 every 8 connection(s)
I20260619 16:45:05.235929 16802 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:45:05.239023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 16802
I20260619 16:45:05.239549 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44505
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46833
--webserver_port=34003
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:45:05.241937 16870 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:45:05.244154 16870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad: Bootstrap starting.
I20260619 16:45:05.264494 16870 log.cc:824] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:05.277899 16870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:45:05.278402 16870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad: Bootstrap complete.
I20260619 16:45:05.282714 16870 raft_consensus.cc:348] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90ee4f9a07e4d23b0b9a9906cb04aad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46833 } }
I20260619 16:45:05.283157 16870 raft_consensus.cc:729] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e90ee4f9a07e4d23b0b9a9906cb04aad, State: Initialized, Role: FOLLOWER
I20260619 16:45:05.283766 16870 consensus_queue.cc:260] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90ee4f9a07e4d23b0b9a9906cb04aad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46833 } }
I20260619 16:45:05.283901 16870 raft_consensus.cc:388] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:45:05.284003 16870 raft_consensus.cc:482] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:45:05.284121 16870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [term 2 FOLLOWER]: Advancing to term 3
I20260619 16:45:05.286371 16870 raft_consensus.cc:504] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90ee4f9a07e4d23b0b9a9906cb04aad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46833 } }
I20260619 16:45:05.286754 16870 leader_election.cc:302] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e90ee4f9a07e4d23b0b9a9906cb04aad; no voters:
I20260619 16:45:05.287129 16870 leader_election.cc:288] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 16:45:05.287218 16874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [term 3 FOLLOWER]: Leader election won for term 3
I20260619 16:45:05.287513 16874 raft_consensus.cc:686] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [term 3 LEADER]: Becoming Leader. State: Replica: e90ee4f9a07e4d23b0b9a9906cb04aad, State: Running, Role: LEADER
I20260619 16:45:05.287971 16874 consensus_queue.cc:237] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90ee4f9a07e4d23b0b9a9906cb04aad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46833 } }
I20260619 16:45:05.288560 16870 sys_catalog.cc:564] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:45:05.289057 16874 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [sys.catalog]: SysCatalogTable state changed. Reason: New leader e90ee4f9a07e4d23b0b9a9906cb04aad. Latest consensus state: current_term: 3 leader_uuid: "e90ee4f9a07e4d23b0b9a9906cb04aad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90ee4f9a07e4d23b0b9a9906cb04aad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46833 } } }
I20260619 16:45:05.289315 16874 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:05.289505 16875 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e90ee4f9a07e4d23b0b9a9906cb04aad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90ee4f9a07e4d23b0b9a9906cb04aad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46833 } } }
I20260619 16:45:05.289806 16875 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:05.290244 16880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:45:05.296088 16880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:45:05.296968 16880 catalog_manager.cc:1259] Loaded cluster ID: 2ad0be05d5354c1f96ef6fd1f980397a
I20260619 16:45:05.297045 16880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:45:05.299067 16880 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:45:05.299822 16880 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad: Loaded TSK: 0
I20260619 16:45:05.299945 16880 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e90ee4f9a07e4d23b0b9a9906cb04aad: Loaded TSK: 1
F20260619 16:45:05.300617 16880 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe4eeaa99ad google::LogMessage::Fail() at ??:0
@ 0x7fe4eeaac88a google::LogMessage::SendToLog() at ??:0
@ 0x7fe4eeaa94de google::LogMessage::Flush() at ??:0
@ 0x7fe4eeaaaa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe4f176d18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe4f1765f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe4f940e7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe4f93e899c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe4f93e9178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fe4f942adf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fe4f944982b std::function<>::operator()() at ??:0
@ 0x7fe4f93e8b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fe4f93e9f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe4f93e6579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fe4f942a640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe4f0cfb3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe4f0cfbc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fe4f0cfd3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe4f0cec512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe4f23306db start_thread at ??:0
@ 0x7fe4ed5a761f clone at ??:0
W20260619 16:45:05.430097 16872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:05.430508 16872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:05.430570 16872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:05.435905 16872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:05.436028 16872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:05.436160 16872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:45:05.443156 16872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34003
--tserver_master_addrs=127.0.98.62:46833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:05.444824 16872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:05.446295 16872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:05.459324 16918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:05.459383 16917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:05.459532 16920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:05.459620 16872 server_base.cc:1034] running on GCE node
I20260619 16:45:05.460485 16872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:05.461383 16872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:05.462579 16872 hybrid_clock.cc:648] HybridClock initialized: now 1781887505462548 us; error 58 us; skew 500 ppm
Jun 19 16:45:05 dist-test-slave-03cd krb5kdc[15725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:05.466399 16872 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:45:05.467906 16872 webserver.cc:460] Webserver started at http://127.0.98.1:34003/ using document root <none> and password file <none>
I20260619 16:45:05.468686 16872 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:05.468796 16872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:05.474351 16872 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:45:05.477643 16927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:05.479511 16872 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260619 16:45:05.479718 16872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "869d5f0889b5494d8fb502c4b810df76"
format_stamp: "Formatted at 2026-06-19 16:44:22 on dist-test-slave-03cd"
server_key: "57fb264417f98af538e63f3865d456eb"
server_key_iv: "bcdb807774b4ffd66a31df8714c9f22b"
server_key_version: "encryptionkey@0"
I20260619 16:45:05.480719 16872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:05.507746 16872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:05.512058 16872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:05.512446 16872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:05.513437 16872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:45:05.515053 16934 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 16:45:05.521443 16872 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 16:45:05.521584 16872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260619 16:45:05.521711 16872 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 16:45:05.523985 16872 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 16:45:05.524077 16872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260619 16:45:05.524586 16934 tablet_bootstrap.cc:492] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76: Bootstrap starting.
I20260619 16:45:05.535897 16872 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44505
I20260619 16:45:05.536047 17041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44505 every 8 connection(s)
I20260619 16:45:05.537263 16872 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:45:05.540104 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 16872
I20260619 16:45:05.540690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:32803
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46833
--webserver_port=44013
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:45:05.542096 17044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:05.540002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.1:36877)
0619 16:45:05.541225 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:05.541335 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1022,"thread_start_us":152,"threads_started":1}
W20260619 16:45:05.542562 17042 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46833 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46833: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
I20260619 16:45:05.543530 17044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:05.543065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.1:54527)
0619 16:45:05.543365 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:05.543436 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":88}
I20260619 16:45:05.545074 17044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:05.544601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.1:57565)
0619 16:45:05.544829 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:05.544929 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260619 16:45:05.545373 17042 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 16:45:05.550741 16934 log.cc:824] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:05.559844 16934 tablet_bootstrap.cc:492] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:45:05.560365 16934 tablet_bootstrap.cc:492] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76: Bootstrap complete.
I20260619 16:45:05.561697 16934 ts_tablet_manager.cc:1397] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76: Time spent bootstrapping tablet: real 0.037s user 0.022s sys 0.013s
I20260619 16:45:05.565095 16934 raft_consensus.cc:348] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 } } peers { permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 } }
I20260619 16:45:05.565624 16934 raft_consensus.cc:729] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869d5f0889b5494d8fb502c4b810df76, State: Initialized, Role: FOLLOWER
I20260619 16:45:05.566265 16934 consensus_queue.cc:260] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 } } peers { permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 } }
I20260619 16:45:05.567605 16934 ts_tablet_manager.cc:1428] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260619 16:45:05.568146 17044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:05.567717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.1:50581)
0619 16:45:05.567981 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:05.568056 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260619 16:45:05.568974 17044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:05.568729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.1:48831)
0619 16:45:05.568846 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:05.568913 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 16:45:05.666385 17045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:05.666706 17045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:05.666757 17045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:05.671057 17045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:05.671149 17045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:05.671214 17045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:45:05.677014 17045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:32803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44013
--tserver_master_addrs=127.0.98.62:46833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:05.678309 17045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:05.679471 17045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:05.691365 17053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:05.691604 17055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:05.691843 17045 server_base.cc:1034] running on GCE node
W20260619 16:45:05.691488 17052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:05.692493 17045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:05.693210 17045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:05.694449 17045 hybrid_clock.cc:648] HybridClock initialized: now 1781887505694414 us; error 64 us; skew 500 ppm
Jun 19 16:45:05 dist-test-slave-03cd krb5kdc[15725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:05.697521 17045 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:45:05.698693 17045 webserver.cc:460] Webserver started at http://127.0.98.2:44013/ using document root <none> and password file <none>
I20260619 16:45:05.699316 17045 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:05.699385 17045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:05.704071 17045 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:45:05.706868 17062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:05.708670 17045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:45:05.708848 17045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "53176a8d1367494f9eeb38acefecfa0c"
format_stamp: "Formatted at 2026-06-19 16:44:23 on dist-test-slave-03cd"
server_key: "18caedd2a847cdea4ba46431788b9649"
server_key_iv: "a6743a74054908d353ad32c88c9f0e31"
server_key_version: "encryptionkey@0"
I20260619 16:45:05.709460 17045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:05.721645 17045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:05.725284 17045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:05.725505 17045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:05.726164 17045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:45:05.727101 17045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:45:05.727155 17045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:05.727198 17045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:45:05.727214 17045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:05.737998 17045 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32803
I20260619 16:45:05.738068 17175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32803 every 8 connection(s)
I20260619 16:45:05.739243 17045 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:45:05.740406 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 17045
I20260619 16:45:05.740906 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43281
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46833
--webserver_port=45957
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:45:05.745342 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:05.742501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.2:36529)
0619 16:45:05.743193 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:05.744076 (+ 883us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":132,"threads_started":1}
W20260619 16:45:05.746006 17176 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46833 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46833: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
I20260619 16:45:05.747082 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:05.746638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.2:34429)
0619 16:45:05.746847 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:05.746950 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 16:45:05.748178 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:05.747837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.2:52985)
0619 16:45:05.747997 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:05.748075 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 16:45:05.748420 17176 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:45:05.878607 17178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:05.878928 17178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:05.878978 17178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:05.883105 17178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:05.883186 17178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:05.883249 17178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:45:05.889329 17178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45957
--tserver_master_addrs=127.0.98.62:46833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:05.890496 17178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:05.891547 17178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:05.902485 17185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:05.902534 17187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:05.902655 17184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:05.902772 17178 server_base.cc:1034] running on GCE node
I20260619 16:45:05.903258 17178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:05.904033 17178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:05.905238 17178 hybrid_clock.cc:648] HybridClock initialized: now 1781887505905218 us; error 40 us; skew 500 ppm
Jun 19 16:45:05 dist-test-slave-03cd krb5kdc[15725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:05.918294 17178 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:45:05.919523 17178 webserver.cc:460] Webserver started at http://127.0.98.3:45957/ using document root <none> and password file <none>
I20260619 16:45:05.920226 17178 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:05.920300 17178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:05.924837 17178 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:45:05.927546 17194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:05.929312 17178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:45:05.929461 17178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "18ec02b6990242519665eb24c9fa5f14"
format_stamp: "Formatted at 2026-06-19 16:44:23 on dist-test-slave-03cd"
server_key: "0907382f51814a0c728ce713b280cb08"
server_key_iv: "ff78012f373e85958ca966a514180804"
server_key_version: "encryptionkey@0"
I20260619 16:45:05.930295 17178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:05.954589 17178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:05.958437 17178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:05.958624 17178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:05.959263 17178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:45:05.960363 17178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:45:05.960431 17178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:05.960510 17178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:45:05.960544 17178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:05.970709 17178 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43281
I20260619 16:45:05.970799 17307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43281 every 8 connection(s)
I20260619 16:45:05.971962 17178 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:45:05.976189 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:05.974300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.3:57973)
0619 16:45:05.974810 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:05.975329 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":137,"threads_started":1}
W20260619 16:45:05.976850 17308 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46833 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46833: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
I20260619 16:45:05.977519 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:05.977283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.3:43195)
0619 16:45:05.977414 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:05.977464 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 16:45:05.978235 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:05.977992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.3:43719)
0619 16:45:05.978133 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:05.978182 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260619 16:45:05.978569 17308 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 16:45:05.980827 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 17178
I20260619 16:45:06.570519 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:06.570026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.1:49711)
0619 16:45:06.570354 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:06.570417 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":80,"threads_started":1}
I20260619 16:45:06.749644 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:06.749173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.2:38583)
0619 16:45:06.749446 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:06.749511 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":80,"threads_started":1}
I20260619 16:45:06.979852 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:06.979282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.3:49505)
0619 16:45:06.979594 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:06.979719 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":94,"threads_started":1}
I20260619 16:45:07.041254 17315 raft_consensus.cc:482] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:45:07.041539 17315 raft_consensus.cc:504] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 } } peers { permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 } }
I20260619 16:45:07.042713 17315 leader_election.cc:288] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 18ec02b6990242519665eb24c9fa5f14 (127.0.98.3:43281), 53176a8d1367494f9eeb38acefecfa0c (127.0.98.2:32803)
Jun 19 16:45:07 dist-test-slave-03cd krb5kdc[15725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 19 16:45:07 dist-test-slave-03cd krb5kdc[15725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260619 16:45:07.061002 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:07.042995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43281 (local address 127.0.98.1:42299)
0619 16:45:07.043442 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:07.043457 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:07.043720 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:07.045610 (+ 1890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:07.045620 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:07.046310 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:45:07.047163 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:07.047185 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:07.049154 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:07.049158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:07.050813 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:07.050827 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:07.051875 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:07.051879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:07.052237 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:07.053190 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:45:07.053227 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:45:07.055987 (+ 2760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:45:07.058484 (+ 2497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:45:07.058490 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:45:07.058492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:45:07.058695 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:45:07.059052 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:45:07.059056 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:45:07.059058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:45:07.059143 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:45:07.059880 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:45:07.059884 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:45:07.060175 (+ 291us) client_negotiation.cc:769] Sending connection context
0619 16:45:07.060384 (+ 209us) client_negotiation.cc:240] Negotiation successful
0619 16:45:07.060794 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":646,"spinlock_wait_cycles":40960,"thread_start_us":69,"threads_started":1}
I20260619 16:45:07.061022 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:07.043170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42299 (local address 127.0.98.3:43281)
0619 16:45:07.043576 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:07.043599 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:07.043642 (+ 43us) server_negotiation.cc:407] Connection header received
0619 16:45:07.044787 (+ 1145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:07.044794 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:07.045180 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:07.045497 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:07.047373 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:07.048966 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:07.050967 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:07.051724 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:07.051901 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:07.056223 (+ 4322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:07.056228 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:07.056238 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:07.056270 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:07.058348 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:07.058824 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:07.058829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:07.058835 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:07.058953 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:07.059236 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:07.059240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:07.059242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:07.059757 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:07.059901 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:07.060498 (+ 597us) server_negotiation.cc:299] Negotiation successful
0619 16:45:07.060782 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":82,"threads_started":1}
I20260619 16:45:07.061035 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:07.042995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32803 (local address 127.0.98.1:49987)
0619 16:45:07.043127 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:07.043160 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:07.043720 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:07.045610 (+ 1890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:07.045620 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:07.046310 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:45:07.047163 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:07.047185 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:07.049154 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:07.049158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:07.050813 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:07.050827 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:07.051875 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:07.051879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:07.052237 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:07.053190 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:45:07.053222 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:45:07.055547 (+ 2325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:45:07.058259 (+ 2712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:45:07.058267 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:45:07.058282 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:45:07.058588 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:45:07.058961 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:45:07.058964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:45:07.058967 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:45:07.059118 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:45:07.059874 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:45:07.059881 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:45:07.060175 (+ 294us) client_negotiation.cc:769] Sending connection context
0619 16:45:07.060383 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 16:45:07.060884 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":561,"spinlock_wait_cycles":54912}
I20260619 16:45:07.061343 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:07.043170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49987 (local address 127.0.98.2:32803)
0619 16:45:07.043576 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:07.043599 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:07.043642 (+ 43us) server_negotiation.cc:407] Connection header received
0619 16:45:07.044764 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:07.044773 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:07.045181 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:07.045497 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:07.047370 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:07.048965 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:07.050967 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:07.051724 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:07.051901 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:07.055911 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:07.055923 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:07.055943 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:07.055984 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:07.058106 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:07.058719 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:07.058726 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:07.058731 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:07.058862 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:07.059215 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:07.059220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:07.059222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:07.059756 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:07.059901 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:07.060494 (+ 593us) server_negotiation.cc:299] Negotiation successful
0619 16:45:07.060782 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":116,"threads_started":1}
W20260619 16:45:07.064527 16931 leader_election.cc:341] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 53176a8d1367494f9eeb38acefecfa0c (127.0.98.2:32803): Not found: Tablet not found: 3b7964a1aaf34e2086ccff174aa0f865
W20260619 16:45:07.065024 16930 leader_election.cc:341] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 18ec02b6990242519665eb24c9fa5f14 (127.0.98.3:43281): Not found: Tablet not found: 3b7964a1aaf34e2086ccff174aa0f865
I20260619 16:45:07.065112 16930 leader_election.cc:302] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869d5f0889b5494d8fb502c4b810df76; no voters: 18ec02b6990242519665eb24c9fa5f14, 53176a8d1367494f9eeb38acefecfa0c
I20260619 16:45:07.065357 17315 raft_consensus.cc:2726] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:45:07.571861 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:07.571325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.1:59623)
0619 16:45:07.571613 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:07.571752 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":88,"threads_started":1}
I20260619 16:45:07.751012 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:07.750528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.2:37121)
0619 16:45:07.750828 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:07.750911 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":83,"threads_started":1}
I20260619 16:45:07.981853 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:07.980933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.3:58837)
0619 16:45:07.981497 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:07.981618 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":91,"threads_started":1}
I20260619 16:45:08.573336 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:08.572793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.1:43567)
0619 16:45:08.573166 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:08.573227 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":96,"threads_started":1}
I20260619 16:45:08.752774 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:08.752148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.2:47479)
0619 16:45:08.752560 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:08.752646 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":114,"threads_started":1}
I20260619 16:45:08.983641 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:08.983066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.3:42183)
0619 16:45:08.983405 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:08.983477 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":91,"threads_started":1}
I20260619 16:45:09.184074 17325 raft_consensus.cc:482] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:45:09.184247 17325 raft_consensus.cc:504] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 } } peers { permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 } }
I20260619 16:45:09.184785 17325 leader_election.cc:288] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 18ec02b6990242519665eb24c9fa5f14 (127.0.98.3:43281), 53176a8d1367494f9eeb38acefecfa0c (127.0.98.2:32803)
W20260619 16:45:09.186312 16930 leader_election.cc:341] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 18ec02b6990242519665eb24c9fa5f14 (127.0.98.3:43281): Not found: Tablet not found: 3b7964a1aaf34e2086ccff174aa0f865
W20260619 16:45:09.186447 16931 leader_election.cc:341] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 53176a8d1367494f9eeb38acefecfa0c (127.0.98.2:32803): Not found: Tablet not found: 3b7964a1aaf34e2086ccff174aa0f865
I20260619 16:45:09.186650 16931 leader_election.cc:302] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869d5f0889b5494d8fb502c4b810df76; no voters: 18ec02b6990242519665eb24c9fa5f14, 53176a8d1367494f9eeb38acefecfa0c
I20260619 16:45:09.186856 17325 raft_consensus.cc:2726] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:45:09.575070 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:09.574480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.1:58623)
0619 16:45:09.574858 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:09.574941 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":117,"threads_started":1}
I20260619 16:45:09.754251 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:09.753739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.2:54685)
0619 16:45:09.754059 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:09.754125 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":101,"threads_started":1}
I20260619 16:45:09.985478 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:09.984878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.3:39289)
0619 16:45:09.985224 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:09.985318 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":88,"threads_started":1}
I20260619 16:45:10.577276 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:10.576520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.1:44037)
0619 16:45:10.576968 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:10.577071 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":99,"threads_started":1}
I20260619 16:45:10.756110 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:10.755396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.2:44107)
0619 16:45:10.755908 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:10.755993 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":111,"threads_started":1}
I20260619 16:45:10.987205 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:10.986620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.3:34797)
0619 16:45:10.986953 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:10.987039 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":117,"threads_started":1}
I20260619 16:45:11.579041 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:11.578401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.1:51423)
0619 16:45:11.578827 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:11.578897 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":124,"threads_started":1}
I20260619 16:45:11.757511 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:11.757075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.2:52471)
0619 16:45:11.757365 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:11.757417 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":130,"threads_started":1}
I20260619 16:45:11.988770 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:11.988310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.3:59777)
0619 16:45:11.988609 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:11.988673 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":81,"threads_started":1}
I20260619 16:45:12.520845 17335 raft_consensus.cc:482] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:45:12.521013 17335 raft_consensus.cc:504] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 } } peers { permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 } }
I20260619 16:45:12.521525 17335 leader_election.cc:288] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 18ec02b6990242519665eb24c9fa5f14 (127.0.98.3:43281), 53176a8d1367494f9eeb38acefecfa0c (127.0.98.2:32803)
W20260619 16:45:12.522595 16931 leader_election.cc:341] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 53176a8d1367494f9eeb38acefecfa0c (127.0.98.2:32803): Not found: Tablet not found: 3b7964a1aaf34e2086ccff174aa0f865
W20260619 16:45:12.522722 16930 leader_election.cc:341] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 18ec02b6990242519665eb24c9fa5f14 (127.0.98.3:43281): Not found: Tablet not found: 3b7964a1aaf34e2086ccff174aa0f865
I20260619 16:45:12.522876 16930 leader_election.cc:302] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869d5f0889b5494d8fb502c4b810df76; no voters: 18ec02b6990242519665eb24c9fa5f14, 53176a8d1367494f9eeb38acefecfa0c
I20260619 16:45:12.523067 17335 raft_consensus.cc:2726] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:45:12.580808 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:12.580202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.1:47197)
0619 16:45:12.580599 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:12.580686 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":129,"threads_started":1}
I20260619 16:45:12.759193 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:12.758403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.2:43531)
0619 16:45:12.758956 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:12.759056 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":79,"threads_started":1}
I20260619 16:45:12.990767 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:12.990005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.3:40049)
0619 16:45:12.990535 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:12.990633 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":135,"threads_started":1}
I20260619 16:45:13.582556 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:13.581879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.1:35093)
0619 16:45:13.582354 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:13.582444 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":135,"threads_started":1}
I20260619 16:45:13.760927 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:13.760328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.2:40775)
0619 16:45:13.760735 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:13.760809 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":121,"threads_started":1}
I20260619 16:45:13.992671 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:13.992038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.3:55149)
0619 16:45:13.992454 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:13.992545 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":116,"threads_started":1}
I20260619 16:45:14.584335 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:14.583720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.1:38467)
0619 16:45:14.584092 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:14.584184 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":84,"threads_started":1}
I20260619 16:45:14.762315 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:14.761845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.2:54585)
0619 16:45:14.762153 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:14.762214 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":127,"threads_started":1}
I20260619 16:45:14.832712 17344 raft_consensus.cc:482] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:45:14.832868 17344 raft_consensus.cc:504] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 } } peers { permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 } }
I20260619 16:45:14.833631 17344 leader_election.cc:288] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 18ec02b6990242519665eb24c9fa5f14 (127.0.98.3:43281), 53176a8d1367494f9eeb38acefecfa0c (127.0.98.2:32803)
W20260619 16:45:14.834497 16930 leader_election.cc:341] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 18ec02b6990242519665eb24c9fa5f14 (127.0.98.3:43281): Not found: Tablet not found: 3b7964a1aaf34e2086ccff174aa0f865
W20260619 16:45:14.834712 16931 leader_election.cc:341] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 53176a8d1367494f9eeb38acefecfa0c (127.0.98.2:32803): Not found: Tablet not found: 3b7964a1aaf34e2086ccff174aa0f865
I20260619 16:45:14.834794 16931 leader_election.cc:302] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869d5f0889b5494d8fb502c4b810df76; no voters: 18ec02b6990242519665eb24c9fa5f14, 53176a8d1367494f9eeb38acefecfa0c
I20260619 16:45:14.834956 17344 raft_consensus.cc:2726] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:45:14.994129 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:14.993685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.3:34045)
0619 16:45:14.993968 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:14.994031 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":94,"threads_started":1}
I20260619 16:45:15.586041 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:15.585381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.1:34639)
0619 16:45:15.585794 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:15.585901 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":90,"threads_started":1}
I20260619 16:45:15.763971 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:15.763383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.2:35919)
0619 16:45:15.763796 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:15.763858 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":124,"threads_started":1}
I20260619 16:45:15.995893 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:15.995153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.3:51555)
0619 16:45:15.995607 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:15.995780 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":122,"threads_started":1}
I20260619 16:45:16.532615 17349 raft_consensus.cc:482] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:45:16.532807 17349 raft_consensus.cc:504] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 } } peers { permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 } }
I20260619 16:45:16.533309 17349 leader_election.cc:288] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 18ec02b6990242519665eb24c9fa5f14 (127.0.98.3:43281), 53176a8d1367494f9eeb38acefecfa0c (127.0.98.2:32803)
W20260619 16:45:16.534394 16930 leader_election.cc:341] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 18ec02b6990242519665eb24c9fa5f14 (127.0.98.3:43281): Not found: Tablet not found: 3b7964a1aaf34e2086ccff174aa0f865
W20260619 16:45:16.534538 16931 leader_election.cc:341] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 53176a8d1367494f9eeb38acefecfa0c (127.0.98.2:32803): Not found: Tablet not found: 3b7964a1aaf34e2086ccff174aa0f865
I20260619 16:45:16.534718 16931 leader_election.cc:302] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869d5f0889b5494d8fb502c4b810df76; no voters: 18ec02b6990242519665eb24c9fa5f14, 53176a8d1367494f9eeb38acefecfa0c
I20260619 16:45:16.534947 17349 raft_consensus.cc:2726] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:45:16.587604 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:16.586991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.1:37817)
0619 16:45:16.587344 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:16.587459 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":85,"threads_started":1}
I20260619 16:45:16.765455 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:16.764900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.2:58293)
0619 16:45:16.765243 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:16.765324 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":88,"threads_started":1}
I20260619 16:45:16.997568 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:16.996925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.3:36917)
0619 16:45:16.997354 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:16.997441 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":131,"threads_started":1}
I20260619 16:45:17.589383 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:17.588797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.1:41597)
0619 16:45:17.589183 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:17.589271 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":133,"threads_started":1}
I20260619 16:45:17.767232 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:17.766630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.2:47357)
0619 16:45:17.767005 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:17.767095 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":105,"threads_started":1}
I20260619 16:45:17.999477 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:17.998824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.3:42979)
0619 16:45:17.999262 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:17.999353 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":135,"threads_started":1}
I20260619 16:45:18.591185 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:18.590520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.1:33325)
0619 16:45:18.590952 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:18.591052 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":141,"threads_started":1}
I20260619 16:45:18.769075 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:18.768450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.2:35743)
0619 16:45:18.768843 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:18.768959 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":111,"threads_started":1}
I20260619 16:45:19.001451 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:19.000757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.3:54379)
0619 16:45:19.001159 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:19.001271 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":118,"threads_started":1}
I20260619 16:45:19.593042 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:19.592443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.1:44337)
0619 16:45:19.592795 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:19.592884 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":100,"threads_started":1}
I20260619 16:45:19.770766 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:19.770181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.2:52669)
0619 16:45:19.770570 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:19.770653 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":112,"threads_started":1}
I20260619 16:45:20.003322 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:20.002556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.3:44163)
0619 16:45:20.003065 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:20.003179 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":150,"threads_started":1}
I20260619 16:45:20.414992 17362 raft_consensus.cc:482] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:45:20.415155 17362 raft_consensus.cc:504] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ec02b6990242519665eb24c9fa5f14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43281 } } peers { permanent_uuid: "53176a8d1367494f9eeb38acefecfa0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "869d5f0889b5494d8fb502c4b810df76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44505 } }
I20260619 16:45:20.415588 17362 leader_election.cc:288] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 18ec02b6990242519665eb24c9fa5f14 (127.0.98.3:43281), 53176a8d1367494f9eeb38acefecfa0c (127.0.98.2:32803)
W20260619 16:45:20.416715 16931 leader_election.cc:341] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 53176a8d1367494f9eeb38acefecfa0c (127.0.98.2:32803): Not found: Tablet not found: 3b7964a1aaf34e2086ccff174aa0f865
W20260619 16:45:20.416889 16930 leader_election.cc:341] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 18ec02b6990242519665eb24c9fa5f14 (127.0.98.3:43281): Not found: Tablet not found: 3b7964a1aaf34e2086ccff174aa0f865
I20260619 16:45:20.417030 16930 leader_election.cc:302] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869d5f0889b5494d8fb502c4b810df76; no voters: 18ec02b6990242519665eb24c9fa5f14, 53176a8d1367494f9eeb38acefecfa0c
I20260619 16:45:20.417268 17362 raft_consensus.cc:2726] T 3b7964a1aaf34e2086ccff174aa0f865 P 869d5f0889b5494d8fb502c4b810df76 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:45:20.594970 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:20.594408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.1:50327)
0619 16:45:20.594775 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:20.594859 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":128,"threads_started":1}
I20260619 16:45:20.772620 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:20.771973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46833 (local address 127.0.98.2:48277)
0619 16:45:20.772361 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:20.772430 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":211,"threads_started":1}
I20260619 16:45:20.987933 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 16872
I20260619 16:45:20.995131 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 17045
I20260619 16:45:21.002574 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 17178
2026-06-19T16:45:21Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59724 ms)
[----------] 30 tests from SecurityITest (261421 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:45:21 dist-test-slave-03cd krb5kdc[17367](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:45:21 dist-test-slave-03cd krb5kdc[17367](info): set up 2 sockets
Jun 19 16:45:21 dist-test-slave-03cd krb5kdc[17367](info): commencing operation
krb5kdc: starting...
W20260619 16:45:22.095840 392 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.003s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:45:22 dist-test-slave-03cd krb5kdc[17367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:45:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:45:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:45:22.232043 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33607
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:22.368824 17380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:22.369225 17380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:22.369277 17380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:22.373835 17380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:45:22.373934 17380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:22.373955 17380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:22.373975 17380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:45:22.373991 17380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:45:22.380314 17380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:22.381767 17380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:22.383028 17380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:22.393849 17386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:22.393874 17385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:22.394033 17380 server_base.cc:1034] running on GCE node
W20260619 16:45:22.393874 17388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:22.394878 17380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:22.396293 17380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:22.397533 17380 hybrid_clock.cc:648] HybridClock initialized: now 1781887522397493 us; error 54 us; skew 500 ppm
Jun 19 16:45:22 dist-test-slave-03cd krb5kdc[17367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:22.401170 17380 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:45:22.402308 17380 webserver.cc:460] Webserver started at http://127.0.98.62:35435/ using document root <none> and password file <none>
I20260619 16:45:22.403039 17380 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:22.403151 17380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:22.403404 17380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:22.405855 17380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "1cdeac8cf1e64b208c8a83a182de1f37"
format_stamp: "Formatted at 2026-06-19 16:45:22 on dist-test-slave-03cd"
server_key: "b8a1fe2c592b9288717918992c80b0b5"
server_key_iv: "55a7e0070645a6347838124d02c9cefb"
server_key_version: "encryptionkey@0"
I20260619 16:45:22.406517 17380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "1cdeac8cf1e64b208c8a83a182de1f37"
format_stamp: "Formatted at 2026-06-19 16:45:22 on dist-test-slave-03cd"
server_key: "b8a1fe2c592b9288717918992c80b0b5"
server_key_iv: "55a7e0070645a6347838124d02c9cefb"
server_key_version: "encryptionkey@0"
I20260619 16:45:22.411285 17380 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 16:45:22.414726 17395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:22.416782 17380 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:45:22.417110 17380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "1cdeac8cf1e64b208c8a83a182de1f37"
format_stamp: "Formatted at 2026-06-19 16:45:22 on dist-test-slave-03cd"
server_key: "b8a1fe2c592b9288717918992c80b0b5"
server_key_iv: "55a7e0070645a6347838124d02c9cefb"
server_key_version: "encryptionkey@0"
I20260619 16:45:22.417277 17380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:22.433856 17380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:22.437822 17380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:22.438086 17380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:22.447650 17380 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33607
I20260619 16:45:22.447880 17447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33607 every 8 connection(s)
I20260619 16:45:22.449102 17380 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:45:22.450304 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 17380
I20260619 16:45:22.450506 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:45:22.450826 392 external_mini_cluster.cc:1428] Setting key 928bd4067301b8a25b5332b306aa9a9f
I20260619 16:45:22.453733 17448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:22.463166 17448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cdeac8cf1e64b208c8a83a182de1f37: Bootstrap starting.
Jun 19 16:45:22 dist-test-slave-03cd krb5kdc[17367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:45:22.467212 17448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cdeac8cf1e64b208c8a83a182de1f37: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:22.468541 17448 log.cc:824] T 00000000000000000000000000000000 P 1cdeac8cf1e64b208c8a83a182de1f37: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:22.471499 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:22.452786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38312 (local address 127.0.98.62:33607)
0619 16:45:22.453409 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:22.453434 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:22.453497 (+ 63us) server_negotiation.cc:407] Connection header received
0619 16:45:22.454586 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:22.454628 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:22.455191 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:22.455748 (+ 557us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:22.456635 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.458500 (+ 1865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.459854 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.460708 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.460934 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:22.464280 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:22.464288 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:22.464309 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:22.464361 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:22.466968 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:22.467817 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:22.467825 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:22.467832 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:22.468019 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:22.468378 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:22.468382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:22.468384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:22.469088 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:22.469294 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:22.469755 (+ 461us) server_negotiation.cc:299] Negotiation successful
0619 16:45:22.470366 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":162,"threads_started":1}
I20260619 16:45:22.471730 17448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cdeac8cf1e64b208c8a83a182de1f37: No bootstrap required, opened a new log
I20260619 16:45:22.476583 17448 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cdeac8cf1e64b208c8a83a182de1f37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cdeac8cf1e64b208c8a83a182de1f37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33607 } }
I20260619 16:45:22.476903 17448 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cdeac8cf1e64b208c8a83a182de1f37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:22.476940 17448 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cdeac8cf1e64b208c8a83a182de1f37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cdeac8cf1e64b208c8a83a182de1f37, State: Initialized, Role: FOLLOWER
I20260619 16:45:22.477449 17448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cdeac8cf1e64b208c8a83a182de1f37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cdeac8cf1e64b208c8a83a182de1f37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33607 } }
I20260619 16:45:22.477653 17448 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cdeac8cf1e64b208c8a83a182de1f37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:45:22.477720 17448 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cdeac8cf1e64b208c8a83a182de1f37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:45:22.477823 17448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cdeac8cf1e64b208c8a83a182de1f37 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:22.479243 17448 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cdeac8cf1e64b208c8a83a182de1f37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cdeac8cf1e64b208c8a83a182de1f37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33607 } }
I20260619 16:45:22.479717 17448 leader_election.cc:302] T 00000000000000000000000000000000 P 1cdeac8cf1e64b208c8a83a182de1f37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cdeac8cf1e64b208c8a83a182de1f37; no voters:
I20260619 16:45:22.480119 17448 leader_election.cc:288] T 00000000000000000000000000000000 P 1cdeac8cf1e64b208c8a83a182de1f37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:45:22.480391 17453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cdeac8cf1e64b208c8a83a182de1f37 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:22.480774 17453 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cdeac8cf1e64b208c8a83a182de1f37 [term 1 LEADER]: Becoming Leader. State: Replica: 1cdeac8cf1e64b208c8a83a182de1f37, State: Running, Role: LEADER
I20260619 16:45:22.481241 17453 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cdeac8cf1e64b208c8a83a182de1f37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cdeac8cf1e64b208c8a83a182de1f37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33607 } }
I20260619 16:45:22.481688 17448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cdeac8cf1e64b208c8a83a182de1f37 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:45:22.483275 17455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cdeac8cf1e64b208c8a83a182de1f37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cdeac8cf1e64b208c8a83a182de1f37. Latest consensus state: current_term: 1 leader_uuid: "1cdeac8cf1e64b208c8a83a182de1f37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cdeac8cf1e64b208c8a83a182de1f37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33607 } } }
I20260619 16:45:22.483453 17455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cdeac8cf1e64b208c8a83a182de1f37 [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:22.483621 17454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cdeac8cf1e64b208c8a83a182de1f37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cdeac8cf1e64b208c8a83a182de1f37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cdeac8cf1e64b208c8a83a182de1f37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33607 } } }
I20260619 16:45:22.483779 17454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cdeac8cf1e64b208c8a83a182de1f37 [sys.catalog]: This master's current role is: LEADER
W20260619 16:45:22.487567 17468 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1cdeac8cf1e64b208c8a83a182de1f37: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:45:22.487722 17468 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:45:22.487864 17465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:45:22.489089 17465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:45:22.497140 17465 catalog_manager.cc:1347] Generated new cluster ID: b1eeaad223f54e399f592631bb358632
I20260619 16:45:22.497323 17465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:45:22.523823 17465 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:45:22.525254 17465 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:45:22.534437 17465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cdeac8cf1e64b208c8a83a182de1f37: Generated new TSK 0
I20260619 16:45:22.535373 17465 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:45:22.576058 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33607
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:22.713339 17474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:22.713833 17474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:22.713893 17474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:22.718654 17474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:22.718760 17474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:22.718835 17474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:45:22.725648 17474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:22.727231 17474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:22.728672 17474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:22.741693 17479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:22.741691 17482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:22.741950 17480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:22.742504 17474 server_base.cc:1034] running on GCE node
I20260619 16:45:22.743106 17474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:22.744024 17474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:22.745338 17474 hybrid_clock.cc:648] HybridClock initialized: now 1781887522745313 us; error 48 us; skew 500 ppm
Jun 19 16:45:22 dist-test-slave-03cd krb5kdc[17367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:22.749218 17474 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:45:22.750387 17474 webserver.cc:460] Webserver started at http://127.0.98.1:46175/ using document root <none> and password file <none>
I20260619 16:45:22.751094 17474 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:22.751207 17474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:22.751466 17474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:22.753942 17474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-0/data/instance:
uuid: "d49732709f90494f9d60a1519dc19314"
format_stamp: "Formatted at 2026-06-19 16:45:22 on dist-test-slave-03cd"
server_key: "6662c8dab23b0e1174c2d49b8deac763"
server_key_iv: "d9bd7ebf5faae4bf820e85a05d519b2a"
server_key_version: "encryptionkey@0"
I20260619 16:45:22.754726 17474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d49732709f90494f9d60a1519dc19314"
format_stamp: "Formatted at 2026-06-19 16:45:22 on dist-test-slave-03cd"
server_key: "6662c8dab23b0e1174c2d49b8deac763"
server_key_iv: "d9bd7ebf5faae4bf820e85a05d519b2a"
server_key_version: "encryptionkey@0"
I20260619 16:45:22.759771 17474 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260619 16:45:22.763163 17489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:22.765213 17474 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:45:22.765448 17474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "d49732709f90494f9d60a1519dc19314"
format_stamp: "Formatted at 2026-06-19 16:45:22 on dist-test-slave-03cd"
server_key: "6662c8dab23b0e1174c2d49b8deac763"
server_key_iv: "d9bd7ebf5faae4bf820e85a05d519b2a"
server_key_version: "encryptionkey@0"
I20260619 16:45:22.765615 17474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:22.782536 17474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:22.787458 17474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:22.787788 17474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:22.788540 17474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:45:22.789618 17474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:45:22.789738 17474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:22.789811 17474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:45:22.789839 17474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:22.802459 17474 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46029
I20260619 16:45:22.802541 17602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46029 every 8 connection(s)
I20260619 16:45:22.803587 17474 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:45:22.805840 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 17474
I20260619 16:45:22.805981 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:45:22.806396 392 external_mini_cluster.cc:1428] Setting key 4c48e2f09811243b5ee8feb1a7c0ed49
Jun 19 16:45:22 dist-test-slave-03cd krb5kdc[17367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 16:45:22.825160 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:22.806476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38083 (local address 127.0.98.62:33607)
0619 16:45:22.806653 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:22.806658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:22.808475 (+ 1817us) server_negotiation.cc:407] Connection header received
0619 16:45:22.809676 (+ 1201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:22.809679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:22.809751 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:22.809843 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:22.811602 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.812309 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.814857 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.815165 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.815303 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:22.819580 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:22.819585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:22.819588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:22.819677 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:22.821847 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:22.822772 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:22.822777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:22.822779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:22.822911 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:22.823416 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:22.823422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:22.823427 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:22.823817 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:22.823950 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:22.824803 (+ 853us) server_negotiation.cc:299] Negotiation successful
0619 16:45:22.824976 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 16:45:22.826992 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:22.806939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33607 (local address 127.0.98.1:38083)
0619 16:45:22.808240 (+ 1301us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:22.808340 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:22.809407 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:22.810107 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:22.810116 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:22.810602 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:45:22.811357 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:22.811385 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.812511 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.812518 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:22.814610 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:22.814637 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.815446 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.815453 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:22.815801 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:22.816735 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:45:22.816766 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:45:22.819143 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:45:22.822074 (+ 2931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:45:22.822085 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:45:22.822112 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:45:22.822621 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:45:22.823073 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:45:22.823077 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:45:22.823078 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:45:22.823275 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:45:22.824022 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:45:22.824030 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:45:22.824465 (+ 435us) client_negotiation.cc:769] Sending connection context
0619 16:45:22.824708 (+ 243us) client_negotiation.cc:240] Negotiation successful
0619 16:45:22.825268 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"thread_start_us":189,"threads_started":1}
I20260619 16:45:22.829319 17603 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33607
I20260619 16:45:22.829667 17603 heartbeater.cc:464] Registering TS with master...
I20260619 16:45:22.830293 17603 heartbeater.cc:511] Master 127.0.98.62:33607 requested a full tablet report, sending...
I20260619 16:45:22.832302 17412 ts_manager.cc:194] Registered new tserver with Master: d49732709f90494f9d60a1519dc19314 (127.0.98.1:46029)
I20260619 16:45:22.833563 17412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:45:22.845119 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33607
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:22.990235 17608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:22.990613 17608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:22.990705 17608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:22.995744 17608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:22.995909 17608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:22.996088 17608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:45:23.004055 17608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:23.005661 17608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:23.007098 17608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:23.021272 17614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:23.021257 17613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:23.021250 17616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:23.021605 17608 server_base.cc:1034] running on GCE node
I20260619 16:45:23.022301 17608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:23.023308 17608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:23.024567 17608 hybrid_clock.cc:648] HybridClock initialized: now 1781887523024521 us; error 65 us; skew 500 ppm
Jun 19 16:45:23 dist-test-slave-03cd krb5kdc[17367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:23.028661 17608 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:45:23.029996 17608 webserver.cc:460] Webserver started at http://127.0.98.2:40949/ using document root <none> and password file <none>
I20260619 16:45:23.030788 17608 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:23.030869 17608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:23.031154 17608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:23.034148 17608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-1/data/instance:
uuid: "1b9cf8baba814ad5b2b4e0f51a99b433"
format_stamp: "Formatted at 2026-06-19 16:45:23 on dist-test-slave-03cd"
server_key: "e60fdd1d1c83be32739071dbbab1292c"
server_key_iv: "a3dc64c98ea9b4937f9549aa89d2bd0e"
server_key_version: "encryptionkey@0"
I20260619 16:45:23.034927 17608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1b9cf8baba814ad5b2b4e0f51a99b433"
format_stamp: "Formatted at 2026-06-19 16:45:23 on dist-test-slave-03cd"
server_key: "e60fdd1d1c83be32739071dbbab1292c"
server_key_iv: "a3dc64c98ea9b4937f9549aa89d2bd0e"
server_key_version: "encryptionkey@0"
I20260619 16:45:23.040038 17608 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260619 16:45:23.043359 17623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:23.045258 17608 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:45:23.045463 17608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "1b9cf8baba814ad5b2b4e0f51a99b433"
format_stamp: "Formatted at 2026-06-19 16:45:23 on dist-test-slave-03cd"
server_key: "e60fdd1d1c83be32739071dbbab1292c"
server_key_iv: "a3dc64c98ea9b4937f9549aa89d2bd0e"
server_key_version: "encryptionkey@0"
I20260619 16:45:23.045626 17608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:23.074419 17608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:23.078996 17608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:23.079272 17608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:23.080127 17608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:45:23.081182 17608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:45:23.081233 17608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:23.081313 17608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:45:23.081365 17608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:23.094270 17608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40043
I20260619 16:45:23.094306 17736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40043 every 8 connection(s)
I20260619 16:45:23.095489 17608 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:45:23.105206 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 17608
I20260619 16:45:23.105389 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:45:23.105685 392 external_mini_cluster.cc:1428] Setting key cc25f73736a9941859ba5bf1909b0306
Jun 19 16:45:23 dist-test-slave-03cd krb5kdc[17367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:45:23.115101 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:23.098019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36139 (local address 127.0.98.62:33607)
0619 16:45:23.098233 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:23.098240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:23.099588 (+ 1348us) server_negotiation.cc:407] Connection header received
0619 16:45:23.100817 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:23.100821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:23.100895 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:23.101016 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:23.102648 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.103285 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.104767 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.105017 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.105200 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:23.109725 (+ 4525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:23.109734 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:23.109740 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:23.109790 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:23.111841 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:23.112776 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:23.112779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:23.112781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:23.112886 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:23.113462 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:23.113465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:23.113466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:23.113756 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:23.113913 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:23.114606 (+ 693us) server_negotiation.cc:299] Negotiation successful
0619 16:45:23.114817 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 16:45:23.116001 17739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:23.098648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33607 (local address 127.0.98.2:36139)
0619 16:45:23.099350 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:23.099422 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:23.100509 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:23.101258 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:23.101267 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:23.101741 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:45:23.102430 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:23.102448 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.103474 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.103477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:23.104588 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:23.104601 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.105295 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.105301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:23.105681 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:23.106708 (+ 1027us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:45:23.106730 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:45:23.109326 (+ 2596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:45:23.112024 (+ 2698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:45:23.112035 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:45:23.112059 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:45:23.112598 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:45:23.113020 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:45:23.113026 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:45:23.113031 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:45:23.113282 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:45:23.113916 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:45:23.113922 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:45:23.114279 (+ 357us) client_negotiation.cc:769] Sending connection context
0619 16:45:23.114473 (+ 194us) client_negotiation.cc:240] Negotiation successful
0619 16:45:23.114792 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":167,"threads_started":1}
I20260619 16:45:23.117429 17737 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33607
I20260619 16:45:23.117904 17737 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:45:23.118750 17737 heartbeater.cc:511] Master 127.0.98.62:33607 requested a full tablet report, sending...
I20260619 16:45:23.120234 17412 ts_manager.cc:194] Registered new tserver with Master: 1b9cf8baba814ad5b2b4e0f51a99b433 (127.0.98.2:40043)
I20260619 16:45:23.120970 17412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36139
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:45:23.140487 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33607
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:23.289044 17742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:23.289354 17742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:23.289400 17742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:23.293663 17742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:23.293740 17742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:23.293802 17742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:45:23.299516 17742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:23.300745 17742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:23.301844 17742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:23.312677 17747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:23.312690 17750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:23.312690 17748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:23.312952 17742 server_base.cc:1034] running on GCE node
I20260619 16:45:23.313519 17742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:23.314231 17742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:23.315454 17742 hybrid_clock.cc:648] HybridClock initialized: now 1781887523315437 us; error 31 us; skew 500 ppm
Jun 19 16:45:23 dist-test-slave-03cd krb5kdc[17367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:23.318531 17742 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:45:23.319626 17742 webserver.cc:460] Webserver started at http://127.0.98.3:42027/ using document root <none> and password file <none>
I20260619 16:45:23.320305 17742 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:23.320410 17742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:23.320688 17742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:23.323172 17742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-2/data/instance:
uuid: "4748977c1fb74fa7986586a78aafda3f"
format_stamp: "Formatted at 2026-06-19 16:45:23 on dist-test-slave-03cd"
server_key: "71ef62d95552660b460841ba3672b3e3"
server_key_iv: "7190c3c6f5d2837634ce08c1e71d8777"
server_key_version: "encryptionkey@0"
I20260619 16:45:23.323869 17742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4748977c1fb74fa7986586a78aafda3f"
format_stamp: "Formatted at 2026-06-19 16:45:23 on dist-test-slave-03cd"
server_key: "71ef62d95552660b460841ba3672b3e3"
server_key_iv: "7190c3c6f5d2837634ce08c1e71d8777"
server_key_version: "encryptionkey@0"
I20260619 16:45:23.328058 17742 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:45:23.330852 17757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:23.332386 17742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:45:23.332571 17742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "4748977c1fb74fa7986586a78aafda3f"
format_stamp: "Formatted at 2026-06-19 16:45:23 on dist-test-slave-03cd"
server_key: "71ef62d95552660b460841ba3672b3e3"
server_key_iv: "7190c3c6f5d2837634ce08c1e71d8777"
server_key_version: "encryptionkey@0"
I20260619 16:45:23.332703 17742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:23.351048 17742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:23.354612 17742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:23.354944 17742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:23.355624 17742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:45:23.356808 17742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:45:23.356951 17742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:23.357056 17742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:45:23.357110 17742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:23.367547 17742 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41635
I20260619 16:45:23.367564 17870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41635 every 8 connection(s)
I20260619 16:45:23.368747 17742 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:45:23.369627 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 17742
I20260619 16:45:23.369767 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:45:23.370134 392 external_mini_cluster.cc:1428] Setting key 5bc548f37f784c216c226b901c5899c9
Jun 19 16:45:23 dist-test-slave-03cd krb5kdc[17367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:45:23.385215 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:23.371277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56907 (local address 127.0.98.62:33607)
0619 16:45:23.371455 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:23.371459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:23.372885 (+ 1426us) server_negotiation.cc:407] Connection header received
0619 16:45:23.373915 (+ 1030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:23.373918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:23.373971 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:23.374050 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:23.375377 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.376010 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.377269 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.377504 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.377592 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:23.380444 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:23.380448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:23.380450 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:23.380480 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:23.382151 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:23.382917 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:23.382922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:23.382924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:23.383029 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:23.383474 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:23.383494 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:23.383497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:23.383792 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:23.383961 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:23.384856 (+ 895us) server_negotiation.cc:299] Negotiation successful
0619 16:45:23.385050 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:45:23.386036 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:23.371989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33607 (local address 127.0.98.3:56907)
0619 16:45:23.372720 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:23.372777 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:23.373685 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:23.374223 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:23.374231 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:23.374638 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:45:23.375213 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:23.375229 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.376152 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.376155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:23.377139 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:23.377151 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.377658 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.377661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:23.377823 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:23.378516 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:45:23.378534 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:45:23.380230 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:45:23.382345 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:45:23.382351 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:45:23.382366 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:45:23.382790 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:45:23.383143 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:45:23.383151 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:45:23.383152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:45:23.383342 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:45:23.383996 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:45:23.384004 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:45:23.384371 (+ 367us) client_negotiation.cc:769] Sending connection context
0619 16:45:23.384632 (+ 261us) client_negotiation.cc:240] Negotiation successful
0619 16:45:23.385091 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":186,"threads_started":1}
I20260619 16:45:23.387374 17871 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33607
I20260619 16:45:23.387714 17871 heartbeater.cc:464] Registering TS with master...
I20260619 16:45:23.388453 17871 heartbeater.cc:511] Master 127.0.98.62:33607 requested a full tablet report, sending...
I20260619 16:45:23.389865 17412 ts_manager.cc:194] Registered new tserver with Master: 4748977c1fb74fa7986586a78aafda3f (127.0.98.3:41635)
I20260619 16:45:23.390487 17412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56907
I20260619 16:45:23.397704 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:45:23.408603 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:23.400098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38318 (local address 127.0.98.62:33607)
0619 16:45:23.400231 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:23.400235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:23.400368 (+ 133us) server_negotiation.cc:407] Connection header received
0619 16:45:23.400442 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:23.400445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:23.400492 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:23.400574 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:23.401375 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.401929 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.402778 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.403042 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.403141 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:23.404819 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:23.404823 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:23.404825 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:23.404858 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:23.406576 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:23.407180 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:23.407184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:23.407185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:23.407274 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:23.407585 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:23.407587 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:23.407589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:23.407931 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:23.408057 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:23.408327 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 16:45:23.408456 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:45:23.412305 17412 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:45:23.415737 17412 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:45:23.432461 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:23.427110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40262 (local address 127.0.98.3:41635)
0619 16:45:23.427388 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:23.427402 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:23.427558 (+ 156us) server_negotiation.cc:407] Connection header received
0619 16:45:23.427703 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:23.427708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:23.427891 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:23.428026 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:23.428276 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.429018 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.430911 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.431749 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.431879 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:23.432045 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:23.432175 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 16:45:23.432283 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":100,"threads_started":1}
I20260619 16:45:23.432457 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:23.425925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46029 (local address 127.0.0.1:54822)
0619 16:45:23.426310 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:23.426365 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:23.426506 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:23.427771 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:23.427776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:23.427800 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:23.427953 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:23.427959 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.429456 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.429459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:23.431027 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:23.431041 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.432139 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.432142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:23.432220 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:23.432238 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 16:45:23.432311 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 16:45:23.432368 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":111,"thread_start_us":115,"threads_started":1}
I20260619 16:45:23.432456 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:23.426693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41635 (local address 127.0.0.1:40262)
0619 16:45:23.427443 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:23.427456 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:23.427561 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:23.428041 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:23.428044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:23.428063 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:23.428155 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:23.428161 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.429159 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.429164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:23.430782 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:23.430792 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.431879 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.431881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:23.431966 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:23.431999 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 16:45:23.432141 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 16:45:23.432287 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":114,"thread_start_us":95,"threads_started":1}
I20260619 16:45:23.433315 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:23.426085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54822 (local address 127.0.98.1:46029)
0619 16:45:23.427224 (+ 1139us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:23.427232 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:23.427257 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:45:23.427337 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:23.427345 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:23.427562 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:23.427793 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:23.428083 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.429312 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.431171 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.432001 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.432122 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:23.432248 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:23.433039 (+ 791us) server_negotiation.cc:299] Negotiation successful
0619 16:45:23.433148 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1019,"thread_start_us":100,"threads_started":1}
I20260619 16:45:23.434089 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:23.426131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40043 (local address 127.0.0.1:60284)
0619 16:45:23.426817 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:23.426830 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:23.426919 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:23.427950 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:23.427952 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:23.427980 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:23.428084 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:23.428090 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.429459 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.429462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:23.431040 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:23.431047 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.433653 (+ 2606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.433657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:23.433759 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:23.433782 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:45:23.433870 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 16:45:23.433949 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":152,"thread_start_us":99,"threads_started":1}
I20260619 16:45:23.434136 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:23.426772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60284 (local address 127.0.98.2:40043)
0619 16:45:23.427362 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:23.427374 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:23.427398 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:45:23.427481 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:23.427486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:23.427704 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:23.427854 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:23.428209 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.429330 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.432352 (+ 3022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.433521 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.433638 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:23.433779 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:23.433896 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 16:45:23.433988 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":92,"threads_started":1}
I20260619 16:45:23.436555 17537 tablet_service.cc:1467] Processing CreateTablet for tablet 501bc5372a9247b5b3ac877fa8349632 (DEFAULT_TABLE table=test-table [id=10ae58c43dbe4aef9551cd74f5e67e20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:45:23.436556 17805 tablet_service.cc:1467] Processing CreateTablet for tablet 501bc5372a9247b5b3ac877fa8349632 (DEFAULT_TABLE table=test-table [id=10ae58c43dbe4aef9551cd74f5e67e20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:45:23.437812 17537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 501bc5372a9247b5b3ac877fa8349632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:23.437812 17805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 501bc5372a9247b5b3ac877fa8349632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:23.438858 17671 tablet_service.cc:1467] Processing CreateTablet for tablet 501bc5372a9247b5b3ac877fa8349632 (DEFAULT_TABLE table=test-table [id=10ae58c43dbe4aef9551cd74f5e67e20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:45:23.439903 17671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 501bc5372a9247b5b3ac877fa8349632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:23.446200 17890 tablet_bootstrap.cc:492] T 501bc5372a9247b5b3ac877fa8349632 P 4748977c1fb74fa7986586a78aafda3f: Bootstrap starting.
I20260619 16:45:23.448029 17888 tablet_bootstrap.cc:492] T 501bc5372a9247b5b3ac877fa8349632 P d49732709f90494f9d60a1519dc19314: Bootstrap starting.
I20260619 16:45:23.448093 17889 tablet_bootstrap.cc:492] T 501bc5372a9247b5b3ac877fa8349632 P 1b9cf8baba814ad5b2b4e0f51a99b433: Bootstrap starting.
I20260619 16:45:23.448942 17890 tablet_bootstrap.cc:654] T 501bc5372a9247b5b3ac877fa8349632 P 4748977c1fb74fa7986586a78aafda3f: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:23.450330 17888 tablet_bootstrap.cc:654] T 501bc5372a9247b5b3ac877fa8349632 P d49732709f90494f9d60a1519dc19314: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:23.450541 17889 tablet_bootstrap.cc:654] T 501bc5372a9247b5b3ac877fa8349632 P 1b9cf8baba814ad5b2b4e0f51a99b433: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:23.450465 17890 log.cc:824] T 501bc5372a9247b5b3ac877fa8349632 P 4748977c1fb74fa7986586a78aafda3f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:23.451246 17888 log.cc:824] T 501bc5372a9247b5b3ac877fa8349632 P d49732709f90494f9d60a1519dc19314: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:23.451315 17889 log.cc:824] T 501bc5372a9247b5b3ac877fa8349632 P 1b9cf8baba814ad5b2b4e0f51a99b433: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:23.453635 17888 tablet_bootstrap.cc:492] T 501bc5372a9247b5b3ac877fa8349632 P d49732709f90494f9d60a1519dc19314: No bootstrap required, opened a new log
I20260619 16:45:23.453635 17889 tablet_bootstrap.cc:492] T 501bc5372a9247b5b3ac877fa8349632 P 1b9cf8baba814ad5b2b4e0f51a99b433: No bootstrap required, opened a new log
I20260619 16:45:23.453862 17888 ts_tablet_manager.cc:1397] T 501bc5372a9247b5b3ac877fa8349632 P d49732709f90494f9d60a1519dc19314: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 16:45:23.453862 17889 ts_tablet_manager.cc:1397] T 501bc5372a9247b5b3ac877fa8349632 P 1b9cf8baba814ad5b2b4e0f51a99b433: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 16:45:23.454156 17890 tablet_bootstrap.cc:492] T 501bc5372a9247b5b3ac877fa8349632 P 4748977c1fb74fa7986586a78aafda3f: No bootstrap required, opened a new log
I20260619 16:45:23.454439 17890 ts_tablet_manager.cc:1397] T 501bc5372a9247b5b3ac877fa8349632 P 4748977c1fb74fa7986586a78aafda3f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 16:45:23.458133 17889 raft_consensus.cc:348] T 501bc5372a9247b5b3ac877fa8349632 P 1b9cf8baba814ad5b2b4e0f51a99b433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49732709f90494f9d60a1519dc19314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46029 } } peers { permanent_uuid: "4748977c1fb74fa7986586a78aafda3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41635 } } peers { permanent_uuid: "1b9cf8baba814ad5b2b4e0f51a99b433" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40043 } }
I20260619 16:45:23.458488 17889 raft_consensus.cc:374] T 501bc5372a9247b5b3ac877fa8349632 P 1b9cf8baba814ad5b2b4e0f51a99b433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:23.458549 17889 raft_consensus.cc:729] T 501bc5372a9247b5b3ac877fa8349632 P 1b9cf8baba814ad5b2b4e0f51a99b433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b9cf8baba814ad5b2b4e0f51a99b433, State: Initialized, Role: FOLLOWER
I20260619 16:45:23.458494 17888 raft_consensus.cc:348] T 501bc5372a9247b5b3ac877fa8349632 P d49732709f90494f9d60a1519dc19314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49732709f90494f9d60a1519dc19314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46029 } } peers { permanent_uuid: "4748977c1fb74fa7986586a78aafda3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41635 } } peers { permanent_uuid: "1b9cf8baba814ad5b2b4e0f51a99b433" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40043 } }
I20260619 16:45:23.458722 17888 raft_consensus.cc:374] T 501bc5372a9247b5b3ac877fa8349632 P d49732709f90494f9d60a1519dc19314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:23.458777 17888 raft_consensus.cc:729] T 501bc5372a9247b5b3ac877fa8349632 P d49732709f90494f9d60a1519dc19314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d49732709f90494f9d60a1519dc19314, State: Initialized, Role: FOLLOWER
I20260619 16:45:23.459088 17889 consensus_queue.cc:260] T 501bc5372a9247b5b3ac877fa8349632 P 1b9cf8baba814ad5b2b4e0f51a99b433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49732709f90494f9d60a1519dc19314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46029 } } peers { permanent_uuid: "4748977c1fb74fa7986586a78aafda3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41635 } } peers { permanent_uuid: "1b9cf8baba814ad5b2b4e0f51a99b433" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40043 } }
I20260619 16:45:23.459158 17888 consensus_queue.cc:260] T 501bc5372a9247b5b3ac877fa8349632 P d49732709f90494f9d60a1519dc19314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49732709f90494f9d60a1519dc19314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46029 } } peers { permanent_uuid: "4748977c1fb74fa7986586a78aafda3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41635 } } peers { permanent_uuid: "1b9cf8baba814ad5b2b4e0f51a99b433" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40043 } }
I20260619 16:45:23.459745 17890 raft_consensus.cc:348] T 501bc5372a9247b5b3ac877fa8349632 P 4748977c1fb74fa7986586a78aafda3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49732709f90494f9d60a1519dc19314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46029 } } peers { permanent_uuid: "4748977c1fb74fa7986586a78aafda3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41635 } } peers { permanent_uuid: "1b9cf8baba814ad5b2b4e0f51a99b433" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40043 } }
I20260619 16:45:23.460063 17890 raft_consensus.cc:374] T 501bc5372a9247b5b3ac877fa8349632 P 4748977c1fb74fa7986586a78aafda3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:23.460151 17890 raft_consensus.cc:729] T 501bc5372a9247b5b3ac877fa8349632 P 4748977c1fb74fa7986586a78aafda3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4748977c1fb74fa7986586a78aafda3f, State: Initialized, Role: FOLLOWER
I20260619 16:45:23.460197 17888 ts_tablet_manager.cc:1428] T 501bc5372a9247b5b3ac877fa8349632 P d49732709f90494f9d60a1519dc19314: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260619 16:45:23.460197 17889 ts_tablet_manager.cc:1428] T 501bc5372a9247b5b3ac877fa8349632 P 1b9cf8baba814ad5b2b4e0f51a99b433: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20260619 16:45:23.460570 17737 heartbeater.cc:503] Master 127.0.98.62:33607 was elected leader, sending a full tablet report...
I20260619 16:45:23.460570 17603 heartbeater.cc:503] Master 127.0.98.62:33607 was elected leader, sending a full tablet report...
I20260619 16:45:23.460711 17890 consensus_queue.cc:260] T 501bc5372a9247b5b3ac877fa8349632 P 4748977c1fb74fa7986586a78aafda3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49732709f90494f9d60a1519dc19314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46029 } } peers { permanent_uuid: "4748977c1fb74fa7986586a78aafda3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41635 } } peers { permanent_uuid: "1b9cf8baba814ad5b2b4e0f51a99b433" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40043 } }
I20260619 16:45:23.461676 17890 ts_tablet_manager.cc:1428] T 501bc5372a9247b5b3ac877fa8349632 P 4748977c1fb74fa7986586a78aafda3f: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260619 16:45:23.462051 17871 heartbeater.cc:503] Master 127.0.98.62:33607 was elected leader, sending a full tablet report...
I20260619 16:45:23.478339 17894 raft_consensus.cc:482] T 501bc5372a9247b5b3ac877fa8349632 P 1b9cf8baba814ad5b2b4e0f51a99b433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:45:23.478607 17894 raft_consensus.cc:504] T 501bc5372a9247b5b3ac877fa8349632 P 1b9cf8baba814ad5b2b4e0f51a99b433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49732709f90494f9d60a1519dc19314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46029 } } peers { permanent_uuid: "4748977c1fb74fa7986586a78aafda3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41635 } } peers { permanent_uuid: "1b9cf8baba814ad5b2b4e0f51a99b433" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40043 } }
I20260619 16:45:23.479878 17894 leader_election.cc:288] T 501bc5372a9247b5b3ac877fa8349632 P 1b9cf8baba814ad5b2b4e0f51a99b433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d49732709f90494f9d60a1519dc19314 (127.0.98.1:46029), 4748977c1fb74fa7986586a78aafda3f (127.0.98.3:41635)
I20260619 16:45:23.483918 17739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:23.480138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46029 (local address 127.0.98.2:34757)
0619 16:45:23.480270 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:23.480286 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:23.480387 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:23.480751 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:23.480754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:23.480780 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:23.480895 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:23.480900 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.481747 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.481752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:23.482686 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:23.482695 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.483455 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.483459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:23.483563 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:23.483592 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 16:45:23.483698 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 16:45:23.483778 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":18}
I20260619 16:45:23.483918 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:23.480259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34757 (local address 127.0.98.1:46029)
0619 16:45:23.480424 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:23.480429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:23.480444 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:45:23.480503 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:23.480506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:23.480555 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:23.480641 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:23.481008 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.481644 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.482807 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.483335 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.483453 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:23.483520 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:23.483721 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 16:45:23.483793 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 16:45:23.484383 17897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:23.480444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41635 (local address 127.0.98.2:56889)
0619 16:45:23.480743 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:23.480754 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:23.480866 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:23.481154 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:23.481156 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:23.481172 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:23.481291 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:23.481297 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.482120 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.482125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:23.483164 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:23.483192 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.483960 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.483964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:23.484061 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:23.484082 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 16:45:23.484174 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 16:45:23.484245 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":4,"thread_start_us":140,"threads_started":1}
I20260619 16:45:23.484378 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:23.480567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56889 (local address 127.0.98.3:41635)
0619 16:45:23.480683 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:23.480687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:23.480834 (+ 147us) server_negotiation.cc:407] Connection header received
0619 16:45:23.480973 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:23.481014 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:23.481059 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:23.481144 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:23.481395 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.482014 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.483289 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.483823 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.483950 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:23.484018 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:23.484202 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 16:45:23.484274 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:45:23.484875 17557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "501bc5372a9247b5b3ac877fa8349632" candidate_uuid: "1b9cf8baba814ad5b2b4e0f51a99b433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d49732709f90494f9d60a1519dc19314" is_pre_election: true
I20260619 16:45:23.485252 17557 raft_consensus.cc:2454] T 501bc5372a9247b5b3ac877fa8349632 P d49732709f90494f9d60a1519dc19314 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b9cf8baba814ad5b2b4e0f51a99b433 in term 0.
I20260619 16:45:23.485230 17825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "501bc5372a9247b5b3ac877fa8349632" candidate_uuid: "1b9cf8baba814ad5b2b4e0f51a99b433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4748977c1fb74fa7986586a78aafda3f" is_pre_election: true
I20260619 16:45:23.485504 17825 raft_consensus.cc:2454] T 501bc5372a9247b5b3ac877fa8349632 P 4748977c1fb74fa7986586a78aafda3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b9cf8baba814ad5b2b4e0f51a99b433 in term 0.
I20260619 16:45:23.485816 17625 leader_election.cc:302] T 501bc5372a9247b5b3ac877fa8349632 P 1b9cf8baba814ad5b2b4e0f51a99b433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b9cf8baba814ad5b2b4e0f51a99b433, d49732709f90494f9d60a1519dc19314; no voters:
I20260619 16:45:23.486135 17894 raft_consensus.cc:2781] T 501bc5372a9247b5b3ac877fa8349632 P 1b9cf8baba814ad5b2b4e0f51a99b433 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:45:23.486225 17894 raft_consensus.cc:482] T 501bc5372a9247b5b3ac877fa8349632 P 1b9cf8baba814ad5b2b4e0f51a99b433 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:45:23.486312 17894 raft_consensus.cc:3037] T 501bc5372a9247b5b3ac877fa8349632 P 1b9cf8baba814ad5b2b4e0f51a99b433 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:23.487413 17894 raft_consensus.cc:504] T 501bc5372a9247b5b3ac877fa8349632 P 1b9cf8baba814ad5b2b4e0f51a99b433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49732709f90494f9d60a1519dc19314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46029 } } peers { permanent_uuid: "4748977c1fb74fa7986586a78aafda3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41635 } } peers { permanent_uuid: "1b9cf8baba814ad5b2b4e0f51a99b433" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40043 } }
I20260619 16:45:23.487838 17894 leader_election.cc:288] T 501bc5372a9247b5b3ac877fa8349632 P 1b9cf8baba814ad5b2b4e0f51a99b433 [CANDIDATE]: Term 1 election: Requested vote from peers d49732709f90494f9d60a1519dc19314 (127.0.98.1:46029), 4748977c1fb74fa7986586a78aafda3f (127.0.98.3:41635)
I20260619 16:45:23.488246 17557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "501bc5372a9247b5b3ac877fa8349632" candidate_uuid: "1b9cf8baba814ad5b2b4e0f51a99b433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d49732709f90494f9d60a1519dc19314"
I20260619 16:45:23.488395 17557 raft_consensus.cc:3037] T 501bc5372a9247b5b3ac877fa8349632 P d49732709f90494f9d60a1519dc19314 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:23.488408 17825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "501bc5372a9247b5b3ac877fa8349632" candidate_uuid: "1b9cf8baba814ad5b2b4e0f51a99b433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4748977c1fb74fa7986586a78aafda3f"
I20260619 16:45:23.488530 17825 raft_consensus.cc:3037] T 501bc5372a9247b5b3ac877fa8349632 P 4748977c1fb74fa7986586a78aafda3f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:23.489682 17825 raft_consensus.cc:2454] T 501bc5372a9247b5b3ac877fa8349632 P 4748977c1fb74fa7986586a78aafda3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b9cf8baba814ad5b2b4e0f51a99b433 in term 1.
I20260619 16:45:23.489687 17557 raft_consensus.cc:2454] T 501bc5372a9247b5b3ac877fa8349632 P d49732709f90494f9d60a1519dc19314 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b9cf8baba814ad5b2b4e0f51a99b433 in term 1.
I20260619 16:45:23.490021 17625 leader_election.cc:302] T 501bc5372a9247b5b3ac877fa8349632 P 1b9cf8baba814ad5b2b4e0f51a99b433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b9cf8baba814ad5b2b4e0f51a99b433, 4748977c1fb74fa7986586a78aafda3f; no voters:
I20260619 16:45:23.490249 17894 raft_consensus.cc:2781] T 501bc5372a9247b5b3ac877fa8349632 P 1b9cf8baba814ad5b2b4e0f51a99b433 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:23.490504 17894 raft_consensus.cc:686] T 501bc5372a9247b5b3ac877fa8349632 P 1b9cf8baba814ad5b2b4e0f51a99b433 [term 1 LEADER]: Becoming Leader. State: Replica: 1b9cf8baba814ad5b2b4e0f51a99b433, State: Running, Role: LEADER
I20260619 16:45:23.490953 17894 consensus_queue.cc:237] T 501bc5372a9247b5b3ac877fa8349632 P 1b9cf8baba814ad5b2b4e0f51a99b433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49732709f90494f9d60a1519dc19314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46029 } } peers { permanent_uuid: "4748977c1fb74fa7986586a78aafda3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41635 } } peers { permanent_uuid: "1b9cf8baba814ad5b2b4e0f51a99b433" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40043 } }
I20260619 16:45:23.495046 17412 catalog_manager.cc:5495] T 501bc5372a9247b5b3ac877fa8349632 P 1b9cf8baba814ad5b2b4e0f51a99b433 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b9cf8baba814ad5b2b4e0f51a99b433 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1b9cf8baba814ad5b2b4e0f51a99b433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49732709f90494f9d60a1519dc19314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4748977c1fb74fa7986586a78aafda3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b9cf8baba814ad5b2b4e0f51a99b433" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40043 } health_report { overall_health: HEALTHY } } }
I20260619 16:45:23.517227 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:23.512992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60298 (local address 127.0.98.2:40043)
0619 16:45:23.513134 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:23.513138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:23.513198 (+ 60us) server_negotiation.cc:407] Connection header received
0619 16:45:23.513350 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:23.513352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:23.513427 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:23.513523 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:45:23.513757 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.514299 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.515398 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.515632 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.515844 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:23.516184 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:45:23.516654 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:45:23.516750 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:23.516953 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 16:45:23.517018 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:45:23.523835 17557 raft_consensus.cc:1261] T 501bc5372a9247b5b3ac877fa8349632 P d49732709f90494f9d60a1519dc19314 [term 1 FOLLOWER]: Refusing update from remote peer 1b9cf8baba814ad5b2b4e0f51a99b433: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:45:23.523844 17825 raft_consensus.cc:1261] T 501bc5372a9247b5b3ac877fa8349632 P 4748977c1fb74fa7986586a78aafda3f [term 1 FOLLOWER]: Refusing update from remote peer 1b9cf8baba814ad5b2b4e0f51a99b433: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:45:23.524576 17894 consensus_queue.cc:1035] T 501bc5372a9247b5b3ac877fa8349632 P 1b9cf8baba814ad5b2b4e0f51a99b433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4748977c1fb74fa7986586a78aafda3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:45:23.524811 17898 consensus_queue.cc:1035] T 501bc5372a9247b5b3ac877fa8349632 P 1b9cf8baba814ad5b2b4e0f51a99b433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d49732709f90494f9d60a1519dc19314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:45:23.534466 17903 mvcc.cc:205] Tried to move back new op lower bound from 7298611296345501696 to 7298611296222527488. Current Snapshot: MvccSnapshot[applied={T|T < 7298611296345501696 or (T in {7298611296345501696})}]
I20260619 16:45:23.534979 17902 mvcc.cc:205] Tried to move back new op lower bound from 7298611296345501696 to 7298611296222527488. Current Snapshot: MvccSnapshot[applied={T|T < 7298611296345501696}]
I20260619 16:45:23.542832 17412 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:45:23.543079 17412 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:45:23.545866 17412 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1cdeac8cf1e64b208c8a83a182de1f37: Sending DeleteTablet for 3 replicas of tablet 501bc5372a9247b5b3ac877fa8349632
I20260619 16:45:23.546912 17671 tablet_service.cc:1514] Processing DeleteTablet for tablet 501bc5372a9247b5b3ac877fa8349632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:45:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60284
I20260619 16:45:23.546988 17537 tablet_service.cc:1514] Processing DeleteTablet for tablet 501bc5372a9247b5b3ac877fa8349632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:45:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54822
I20260619 16:45:23.547154 17805 tablet_service.cc:1514] Processing DeleteTablet for tablet 501bc5372a9247b5b3ac877fa8349632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:45:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40262
I20260619 16:45:23.547737 17912 tablet_replica.cc:330] T 501bc5372a9247b5b3ac877fa8349632 P 4748977c1fb74fa7986586a78aafda3f: stopping tablet replica
I20260619 16:45:23.548040 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 17474
I20260619 16:45:23.548059 17910 tablet_replica.cc:330] T 501bc5372a9247b5b3ac877fa8349632 P 1b9cf8baba814ad5b2b4e0f51a99b433: stopping tablet replica
I20260619 16:45:23.549299 17910 raft_consensus.cc:2229] T 501bc5372a9247b5b3ac877fa8349632 P 1b9cf8baba814ad5b2b4e0f51a99b433 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:45:23.548161 17912 raft_consensus.cc:2229] T 501bc5372a9247b5b3ac877fa8349632 P 4748977c1fb74fa7986586a78aafda3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:45:23.549912 17910 raft_consensus.cc:2258] T 501bc5372a9247b5b3ac877fa8349632 P 1b9cf8baba814ad5b2b4e0f51a99b433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:45:23.550014 17912 raft_consensus.cc:2258] T 501bc5372a9247b5b3ac877fa8349632 P 4748977c1fb74fa7986586a78aafda3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:45:23.551625 17912 ts_tablet_manager.cc:1905] T 501bc5372a9247b5b3ac877fa8349632 P 4748977c1fb74fa7986586a78aafda3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:45:23.554132 17912 ts_tablet_manager.cc:1918] T 501bc5372a9247b5b3ac877fa8349632 P 4748977c1fb74fa7986586a78aafda3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:45:23.554265 17912 log.cc:1196] T 501bc5372a9247b5b3ac877fa8349632 P 4748977c1fb74fa7986586a78aafda3f: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887259574980-392-0/minicluster-data/ts-2/wal/wals/501bc5372a9247b5b3ac877fa8349632
I20260619 16:45:23.554570 17912 ts_tablet_manager.cc:1939] T 501bc5372a9247b5b3ac877fa8349632 P 4748977c1fb74fa7986586a78aafda3f: Deleting consensus metadata
I20260619 16:45:23.555907 17397 catalog_manager.cc:4841] TS 4748977c1fb74fa7986586a78aafda3f (127.0.98.3:41635): tablet 501bc5372a9247b5b3ac877fa8349632 (table test-table [id=10ae58c43dbe4aef9551cd74f5e67e20]) successfully deleted
W20260619 16:45:23.556257 17397 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46029 (error 108)
I20260619 16:45:23.556457 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 17608
I20260619 16:45:23.558324 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:23.557937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46029 (local address 127.0.0.1:54838)
0619 16:45:23.558121 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:23.558211 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260619 16:45:23.558490 17397 catalog_manager.cc:4584] TS d49732709f90494f9d60a1519dc19314 (127.0.98.1:46029): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 501bc5372a9247b5b3ac877fa8349632: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46029: connect: Connection refused (error 111)
I20260619 16:45:23.563320 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 17742
I20260619 16:45:23.564607 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:23.564194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40043 (local address 127.0.0.1:60308)
0619 16:45:23.564427 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:23.564491 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260619 16:45:23.569420 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 17380
2026-06-19T16:45:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2562 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T16:45:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:45:23Z Disabled control of system clock
I20260619 16:45:23.603874 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40133
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:45:23.732553 17917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:23.732877 17917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:23.732916 17917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:23.737205 17917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:45:23.737288 17917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:23.737304 17917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:23.737323 17917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:45:23.737339 17917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:45:23.742704 17917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40133
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:23.743937 17917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:23.745478 17917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:23.755214 17922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:23.755311 17917 server_base.cc:1034] running on GCE node
W20260619 16:45:23.755270 17923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:23.755330 17925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:23.756197 17917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:23.757436 17917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:23.758621 17917 hybrid_clock.cc:648] HybridClock initialized: now 1781887523758606 us; error 35 us; skew 500 ppm
I20260619 16:45:23.760727 17917 webserver.cc:460] Webserver started at http://127.0.98.62:42107/ using document root <none> and password file <none>
I20260619 16:45:23.761369 17917 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:23.761441 17917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:23.761685 17917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:23.764015 17917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "db9396970c2b47e9ba481ed03f64ae79"
format_stamp: "Formatted at 2026-06-19 16:45:23 on dist-test-slave-03cd"
server_key: "763b99eec6160f3b59d856d8067ebce8"
server_key_iv: "3520885858cd3f79c3446ea79c394f85"
server_key_version: "encryptionkey@0"
I20260619 16:45:23.764645 17917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "db9396970c2b47e9ba481ed03f64ae79"
format_stamp: "Formatted at 2026-06-19 16:45:23 on dist-test-slave-03cd"
server_key: "763b99eec6160f3b59d856d8067ebce8"
server_key_iv: "3520885858cd3f79c3446ea79c394f85"
server_key_version: "encryptionkey@0"
I20260619 16:45:23.769006 17917 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:45:23.772125 17931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:23.773695 17917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:45:23.773890 17917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "db9396970c2b47e9ba481ed03f64ae79"
format_stamp: "Formatted at 2026-06-19 16:45:23 on dist-test-slave-03cd"
server_key: "763b99eec6160f3b59d856d8067ebce8"
server_key_iv: "3520885858cd3f79c3446ea79c394f85"
server_key_version: "encryptionkey@0"
I20260619 16:45:23.774035 17917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:23.800556 17917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:23.801384 17917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:23.801621 17917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:23.810308 17917 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40133
I20260619 16:45:23.810312 17983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40133 every 8 connection(s)
I20260619 16:45:23.812026 17917 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:45:23.816578 17984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:23.821975 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 17917
I20260619 16:45:23.822121 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:45:23.822412 392 external_mini_cluster.cc:1428] Setting key 5c11b3c4ec3c251173f27cf22c5496c2
I20260619 16:45:23.823146 17984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db9396970c2b47e9ba481ed03f64ae79: Bootstrap starting.
I20260619 16:45:23.825734 17984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db9396970c2b47e9ba481ed03f64ae79: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:23.826639 17984 log.cc:824] T 00000000000000000000000000000000 P db9396970c2b47e9ba481ed03f64ae79: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:23.829034 17984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db9396970c2b47e9ba481ed03f64ae79: No bootstrap required, opened a new log
I20260619 16:45:23.832734 17987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:23.824143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39096 (local address 127.0.98.62:40133)
0619 16:45:23.824817 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:23.824831 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:23.824869 (+ 38us) server_negotiation.cc:407] Connection header received
0619 16:45:23.825596 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:23.825614 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:23.826085 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:23.826391 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:23.826830 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.827944 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.829184 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.829734 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.829877 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:23.830811 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:23.830816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:23.830831 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:23.831239 (+ 408us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:23.831323 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:23.831344 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:23.831425 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:23.831595 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 16:45:23.831943 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":153,"threads_started":1}
I20260619 16:45:23.833323 17984 raft_consensus.cc:348] T 00000000000000000000000000000000 P db9396970c2b47e9ba481ed03f64ae79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9396970c2b47e9ba481ed03f64ae79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40133 } }
I20260619 16:45:23.833561 17984 raft_consensus.cc:374] T 00000000000000000000000000000000 P db9396970c2b47e9ba481ed03f64ae79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:23.833619 17984 raft_consensus.cc:729] T 00000000000000000000000000000000 P db9396970c2b47e9ba481ed03f64ae79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db9396970c2b47e9ba481ed03f64ae79, State: Initialized, Role: FOLLOWER
I20260619 16:45:23.834095 17984 consensus_queue.cc:260] T 00000000000000000000000000000000 P db9396970c2b47e9ba481ed03f64ae79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9396970c2b47e9ba481ed03f64ae79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40133 } }
I20260619 16:45:23.834259 17984 raft_consensus.cc:388] T 00000000000000000000000000000000 P db9396970c2b47e9ba481ed03f64ae79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:45:23.834337 17984 raft_consensus.cc:482] T 00000000000000000000000000000000 P db9396970c2b47e9ba481ed03f64ae79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:45:23.834462 17984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db9396970c2b47e9ba481ed03f64ae79 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:23.835438 17984 raft_consensus.cc:504] T 00000000000000000000000000000000 P db9396970c2b47e9ba481ed03f64ae79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9396970c2b47e9ba481ed03f64ae79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40133 } }
I20260619 16:45:23.835852 17984 leader_election.cc:302] T 00000000000000000000000000000000 P db9396970c2b47e9ba481ed03f64ae79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db9396970c2b47e9ba481ed03f64ae79; no voters:
I20260619 16:45:23.836192 17984 leader_election.cc:288] T 00000000000000000000000000000000 P db9396970c2b47e9ba481ed03f64ae79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:45:23.836380 17989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db9396970c2b47e9ba481ed03f64ae79 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:23.836629 17989 raft_consensus.cc:686] T 00000000000000000000000000000000 P db9396970c2b47e9ba481ed03f64ae79 [term 1 LEADER]: Becoming Leader. State: Replica: db9396970c2b47e9ba481ed03f64ae79, State: Running, Role: LEADER
I20260619 16:45:23.836961 17989 consensus_queue.cc:237] T 00000000000000000000000000000000 P db9396970c2b47e9ba481ed03f64ae79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9396970c2b47e9ba481ed03f64ae79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40133 } }
I20260619 16:45:23.837550 17984 sys_catalog.cc:564] T 00000000000000000000000000000000 P db9396970c2b47e9ba481ed03f64ae79 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:45:23.839214 17990 sys_catalog.cc:455] T 00000000000000000000000000000000 P db9396970c2b47e9ba481ed03f64ae79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db9396970c2b47e9ba481ed03f64ae79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9396970c2b47e9ba481ed03f64ae79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40133 } } }
I20260619 16:45:23.839376 17990 sys_catalog.cc:458] T 00000000000000000000000000000000 P db9396970c2b47e9ba481ed03f64ae79 [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:23.839295 17991 sys_catalog.cc:455] T 00000000000000000000000000000000 P db9396970c2b47e9ba481ed03f64ae79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db9396970c2b47e9ba481ed03f64ae79. Latest consensus state: current_term: 1 leader_uuid: "db9396970c2b47e9ba481ed03f64ae79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9396970c2b47e9ba481ed03f64ae79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40133 } } }
I20260619 16:45:23.839457 17991 sys_catalog.cc:458] T 00000000000000000000000000000000 P db9396970c2b47e9ba481ed03f64ae79 [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:23.839820 17995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:45:23.843189 17995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:45:23.849540 17995 catalog_manager.cc:1347] Generated new cluster ID: 278bbc371f6845099adf38b3241f8bd3
I20260619 16:45:23.849673 17995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:45:23.907368 17995 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:45:23.908614 17995 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:45:23.919353 17995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db9396970c2b47e9ba481ed03f64ae79: Generated new TSK 0
I20260619 16:45:23.920406 17995 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:45:23.928763 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40133
--builtin_ntp_servers=127.0.98.20:39569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:45:24.050031 18008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:24.050341 18008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:24.050390 18008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:24.054906 18008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:24.055003 18008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:24.055066 18008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:45:24.060617 18008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:24.061893 18008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:24.063380 18008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:24.074823 18016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:24.074903 18013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:24.074823 18014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:24.075310 18008 server_base.cc:1034] running on GCE node
I20260619 16:45:24.075896 18008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:24.076644 18008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:24.077901 18008 hybrid_clock.cc:648] HybridClock initialized: now 1781887524077868 us; error 46 us; skew 500 ppm
I20260619 16:45:24.080281 18008 webserver.cc:460] Webserver started at http://127.0.98.1:45509/ using document root <none> and password file <none>
I20260619 16:45:24.080936 18008 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:24.081007 18008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:24.081228 18008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:24.083506 18008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-0/data/instance:
uuid: "8b5d17e1cbf44f329b61e8eeb33e0bb4"
format_stamp: "Formatted at 2026-06-19 16:45:24 on dist-test-slave-03cd"
server_key: "ea8ee002983a3d1e738670281cc3e006"
server_key_iv: "c24076c9d073ddc91a8272f868ca58ec"
server_key_version: "encryptionkey@0"
I20260619 16:45:24.084198 18008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8b5d17e1cbf44f329b61e8eeb33e0bb4"
format_stamp: "Formatted at 2026-06-19 16:45:24 on dist-test-slave-03cd"
server_key: "ea8ee002983a3d1e738670281cc3e006"
server_key_iv: "c24076c9d073ddc91a8272f868ca58ec"
server_key_version: "encryptionkey@0"
I20260619 16:45:24.088181 18008 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:45:24.090999 18022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:24.092530 18008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:45:24.092690 18008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "8b5d17e1cbf44f329b61e8eeb33e0bb4"
format_stamp: "Formatted at 2026-06-19 16:45:24 on dist-test-slave-03cd"
server_key: "ea8ee002983a3d1e738670281cc3e006"
server_key_iv: "c24076c9d073ddc91a8272f868ca58ec"
server_key_version: "encryptionkey@0"
I20260619 16:45:24.092829 18008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:24.113829 18008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:24.114614 18008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:24.114845 18008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:24.115482 18008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:45:24.116525 18008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:45:24.116597 18008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:24.116667 18008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:45:24.116706 18008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:24.126453 18008 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39741
I20260619 16:45:24.126482 18135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39741 every 8 connection(s)
I20260619 16:45:24.127418 18008 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:45:24.136873 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 18008
I20260619 16:45:24.137032 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:45:24.137434 392 external_mini_cluster.cc:1428] Setting key c0a4ca28b210173459ac5a0236e9ca2c
I20260619 16:45:24.138126 17987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:24.129423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44613 (local address 127.0.98.62:40133)
0619 16:45:24.129565 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:24.129569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:24.130923 (+ 1354us) server_negotiation.cc:407] Connection header received
0619 16:45:24.132505 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:24.132508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:24.132561 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:24.132646 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:24.133760 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.134399 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.135648 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.135949 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.136036 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:24.137541 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:24.137568 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:24.137575 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:24.137592 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:24.137616 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:24.137626 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:24.137713 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:24.137908 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 16:45:24.137980 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:45:24.139365 18138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:24.129870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40133 (local address 127.0.98.1:44613)
0619 16:45:24.130643 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:24.130693 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:24.132249 (+ 1556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:24.132900 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:24.132908 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:24.133005 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:45:24.133574 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:24.133588 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.134581 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.134585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:24.135517 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:24.135527 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.136097 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.136100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:24.136252 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:24.137183 (+ 931us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:45:24.137185 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:45:24.137231 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:45:24.137304 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:45:24.137728 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:45:24.137744 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:45:24.137751 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 16:45:24.137907 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 16:45:24.138286 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":121,"threads_started":1}
I20260619 16:45:24.140080 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40133
--builtin_ntp_servers=127.0.98.20:39569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:45:24.140945 18136 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40133
I20260619 16:45:24.141255 18136 heartbeater.cc:464] Registering TS with master...
I20260619 16:45:24.141871 18136 heartbeater.cc:511] Master 127.0.98.62:40133 requested a full tablet report, sending...
I20260619 16:45:24.143735 17948 ts_manager.cc:194] Registered new tserver with Master: 8b5d17e1cbf44f329b61e8eeb33e0bb4 (127.0.98.1:39741)
I20260619 16:45:24.145556 17948 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:44613
W20260619 16:45:24.260375 18139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:24.260694 18139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:24.260735 18139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:24.264921 18139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:24.264993 18139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:24.265053 18139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:45:24.270251 18139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:24.271359 18139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:24.272935 18139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:45:24.283635 18139 server_base.cc:1034] running on GCE node
W20260619 16:45:24.283552 18144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:24.283525 18147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:24.283525 18145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:24.284462 18139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:24.285280 18139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:24.286501 18139 hybrid_clock.cc:648] HybridClock initialized: now 1781887524286489 us; error 30 us; skew 500 ppm
I20260619 16:45:24.288544 18139 webserver.cc:460] Webserver started at http://127.0.98.2:42297/ using document root <none> and password file <none>
I20260619 16:45:24.289158 18139 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:24.289253 18139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:24.289495 18139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:24.291877 18139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-1/data/instance:
uuid: "d8ae0b57b4364e66b1e172bb5a859e60"
format_stamp: "Formatted at 2026-06-19 16:45:24 on dist-test-slave-03cd"
server_key: "eb73ea6ae95d635d28a5565d8d299946"
server_key_iv: "ad74bcd787be5b67ee9d1a11b5817684"
server_key_version: "encryptionkey@0"
I20260619 16:45:24.292461 18139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d8ae0b57b4364e66b1e172bb5a859e60"
format_stamp: "Formatted at 2026-06-19 16:45:24 on dist-test-slave-03cd"
server_key: "eb73ea6ae95d635d28a5565d8d299946"
server_key_iv: "ad74bcd787be5b67ee9d1a11b5817684"
server_key_version: "encryptionkey@0"
I20260619 16:45:24.296761 18139 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:45:24.299763 18153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:24.301208 18139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:45:24.301576 18139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "d8ae0b57b4364e66b1e172bb5a859e60"
format_stamp: "Formatted at 2026-06-19 16:45:24 on dist-test-slave-03cd"
server_key: "eb73ea6ae95d635d28a5565d8d299946"
server_key_iv: "ad74bcd787be5b67ee9d1a11b5817684"
server_key_version: "encryptionkey@0"
I20260619 16:45:24.301753 18139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:24.323381 18139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:24.324216 18139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:24.324445 18139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:24.325074 18139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:45:24.326077 18139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:45:24.326151 18139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:24.326226 18139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:45:24.326274 18139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:24.336541 18139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34461
I20260619 16:45:24.336565 18266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34461 every 8 connection(s)
I20260619 16:45:24.337556 18139 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:45:24.338086 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 18139
I20260619 16:45:24.338203 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:45:24.338505 392 external_mini_cluster.cc:1428] Setting key c159c040c3774977028f7c77a703b36c
I20260619 16:45:24.341259 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40133
--builtin_ntp_servers=127.0.98.20:39569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:45:24.349548 17987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:24.340009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46153 (local address 127.0.98.62:40133)
0619 16:45:24.340130 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:24.340135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:24.341655 (+ 1520us) server_negotiation.cc:407] Connection header received
0619 16:45:24.343135 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:24.343138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:24.343197 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:24.343278 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:24.344334 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.345227 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.346662 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.347103 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.347245 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:24.348894 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:24.348900 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:24.348906 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:24.348938 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:24.348966 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:24.348981 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:24.349073 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:24.349237 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 16:45:24.349323 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 16:45:24.350375 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:24.340639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40133 (local address 127.0.98.2:46153)
0619 16:45:24.341470 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:24.341535 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:24.342917 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:24.343461 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:24.343468 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:24.343578 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:45:24.344181 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:24.344195 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.345401 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.345404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:24.346510 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:24.346523 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.347324 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.347332 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:24.347575 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:24.348532 (+ 957us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:45:24.348535 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:45:24.348617 (+ 82us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:45:24.348712 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:45:24.349079 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:45:24.349086 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:45:24.349090 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:45:24.349188 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 16:45:24.349443 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":161,"threads_started":1}
I20260619 16:45:24.351713 18267 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40133
I20260619 16:45:24.352012 18267 heartbeater.cc:464] Registering TS with master...
I20260619 16:45:24.352763 18267 heartbeater.cc:511] Master 127.0.98.62:40133 requested a full tablet report, sending...
I20260619 16:45:24.354003 17948 ts_manager.cc:194] Registered new tserver with Master: d8ae0b57b4364e66b1e172bb5a859e60 (127.0.98.2:34461)
I20260619 16:45:24.354645 17948 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:46153
W20260619 16:45:24.465495 18270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:24.465809 18270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:24.465860 18270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:24.470157 18270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:24.470233 18270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:24.470291 18270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:45:24.476753 18270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:24.478058 18270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:24.479710 18270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:24.490170 18278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:24.490200 18275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:24.490214 18276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:24.490685 18270 server_base.cc:1034] running on GCE node
I20260619 16:45:24.491189 18270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:24.492062 18270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:24.493294 18270 hybrid_clock.cc:648] HybridClock initialized: now 1781887524493240 us; error 66 us; skew 500 ppm
I20260619 16:45:24.495925 18270 webserver.cc:460] Webserver started at http://127.0.98.3:36681/ using document root <none> and password file <none>
I20260619 16:45:24.496667 18270 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:24.496767 18270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:24.497009 18270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:24.499341 18270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-2/data/instance:
uuid: "c88595385b55432e9fb58f5ad52af35e"
format_stamp: "Formatted at 2026-06-19 16:45:24 on dist-test-slave-03cd"
server_key: "7258fa14d072256af52339b1271a0ccd"
server_key_iv: "28d22daef41d894d1b4102e4866d59ac"
server_key_version: "encryptionkey@0"
I20260619 16:45:24.499993 18270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c88595385b55432e9fb58f5ad52af35e"
format_stamp: "Formatted at 2026-06-19 16:45:24 on dist-test-slave-03cd"
server_key: "7258fa14d072256af52339b1271a0ccd"
server_key_iv: "28d22daef41d894d1b4102e4866d59ac"
server_key_version: "encryptionkey@0"
I20260619 16:45:24.504314 18270 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:45:24.507468 18284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:24.509428 18270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:45:24.509601 18270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "c88595385b55432e9fb58f5ad52af35e"
format_stamp: "Formatted at 2026-06-19 16:45:24 on dist-test-slave-03cd"
server_key: "7258fa14d072256af52339b1271a0ccd"
server_key_iv: "28d22daef41d894d1b4102e4866d59ac"
server_key_version: "encryptionkey@0"
I20260619 16:45:24.509747 18270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:24.530979 18270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:24.531912 18270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:24.532172 18270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:24.532903 18270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:45:24.534159 18270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:45:24.534264 18270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:24.534343 18270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:45:24.534396 18270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:24.545478 18270 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41323
I20260619 16:45:24.545497 18397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41323 every 8 connection(s)
I20260619 16:45:24.546538 18270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:45:24.550242 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 18270
I20260619 16:45:24.550369 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:45:24.550670 392 external_mini_cluster.cc:1428] Setting key 5872d03efa580f40df09139b0d3026e7
I20260619 16:45:24.558161 17987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:24.548720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48457 (local address 127.0.98.62:40133)
0619 16:45:24.548863 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:24.548868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:24.550020 (+ 1152us) server_negotiation.cc:407] Connection header received
0619 16:45:24.552343 (+ 2323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:24.552347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:24.552409 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:24.552497 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:24.553654 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.554185 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.555998 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.556227 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.556325 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:24.557634 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:24.557638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:24.557640 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:24.557656 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:24.557677 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:24.557691 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:24.557764 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:24.557959 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 16:45:24.558032 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:45:24.559134 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:24.549065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40133 (local address 127.0.98.3:48457)
0619 16:45:24.549767 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:24.549818 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:24.552081 (+ 2263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:24.552694 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:24.552702 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:24.552845 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:45:24.553471 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:24.553492 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.554474 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.554480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:24.555748 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:24.555768 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.556387 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.556389 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:24.556554 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:24.557288 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:45:24.557290 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:45:24.557336 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:45:24.557418 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:45:24.557782 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:45:24.557788 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:45:24.557793 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:45:24.557913 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 16:45:24.558188 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":134,"threads_started":1}
I20260619 16:45:24.560662 18398 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40133
I20260619 16:45:24.560974 18398 heartbeater.cc:464] Registering TS with master...
I20260619 16:45:24.561602 18398 heartbeater.cc:511] Master 127.0.98.62:40133 requested a full tablet report, sending...
I20260619 16:45:24.562888 17948 ts_manager.cc:194] Registered new tserver with Master: c88595385b55432e9fb58f5ad52af35e (127.0.98.3:41323)
I20260619 16:45:24.563580 17948 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:48457
I20260619 16:45:24.565976 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:45:24.573046 17987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:24.568737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43070 (local address 127.0.98.62:40133)
0619 16:45:24.568891 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:24.568895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:24.569084 (+ 189us) server_negotiation.cc:407] Connection header received
0619 16:45:24.569191 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:24.569194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:24.569236 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:24.569312 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:24.569593 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.570152 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.571055 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.571278 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.571371 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:24.572403 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:24.572407 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:24.572409 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:24.572423 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:24.572442 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:24.572452 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:24.572532 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:24.572840 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 16:45:24.572911 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:45:24.576925 17948 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:45:24.580312 17948 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:45:24.597618 18410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:24.590678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34461 (local address 127.0.0.1:59404)
0619 16:45:24.591492 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:24.591548 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:24.591788 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:24.592617 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:24.592623 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:24.592645 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:24.592781 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:24.592788 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.594483 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.594487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:24.595986 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:24.595993 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.597144 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.597147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:24.597215 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:24.597275 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 16:45:24.597354 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 16:45:24.597430 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":252,"thread_start_us":144,"threads_started":1}
I20260619 16:45:24.597714 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:24.590782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59404 (local address 127.0.98.2:34461)
0619 16:45:24.591221 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:24.591236 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:24.591636 (+ 400us) server_negotiation.cc:407] Connection header received
0619 16:45:24.591994 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:24.592000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:24.592223 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:24.592373 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:24.592920 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.594337 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.596111 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.596723 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.596860 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:24.597113 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:24.597404 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 16:45:24.597531 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":148,"threads_started":1}
I20260619 16:45:24.597908 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:24.590721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41323 (local address 127.0.0.1:38714)
0619 16:45:24.592391 (+ 1670us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:24.592403 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:24.592530 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:24.592935 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:24.592940 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:24.592962 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:24.593049 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:24.593055 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.594491 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.594493 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:24.595804 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:24.595815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.596855 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.596859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:24.597595 (+ 736us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:24.597624 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 16:45:24.597697 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 16:45:24.597773 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1587,"mutex_wait_us":1,"thread_start_us":108,"threads_started":1}
I20260619 16:45:24.598330 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:24.591548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38714 (local address 127.0.98.3:41323)
0619 16:45:24.591975 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:24.591986 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:24.592501 (+ 515us) server_negotiation.cc:407] Connection header received
0619 16:45:24.592640 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:24.592646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:24.592783 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:24.592954 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:24.593138 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.594337 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.595978 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.596710 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.596833 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:24.597056 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:24.598008 (+ 952us) server_negotiation.cc:299] Negotiation successful
0619 16:45:24.598139 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":121,"threads_started":1}
I20260619 16:45:24.598574 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:24.591987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48206 (local address 127.0.98.1:39741)
0619 16:45:24.592282 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:24.592289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:24.592612 (+ 323us) server_negotiation.cc:407] Connection header received
0619 16:45:24.592692 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:24.592696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:24.592890 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:24.593026 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:24.593377 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.594175 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.595782 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.596710 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.596833 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:24.597055 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:24.597320 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 16:45:24.597437 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":130,"threads_started":1}
I20260619 16:45:24.598596 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:24.591435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39741 (local address 127.0.0.1:48206)
0619 16:45:24.592401 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:24.592415 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:24.592504 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:24.593172 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:24.593174 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:24.593195 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:24.593251 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:24.593255 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.594341 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.594345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:24.595596 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:24.595607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.596998 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.597001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:24.597085 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:24.597133 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 16:45:24.597251 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 16:45:24.597412 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"mutex_wait_us":3,"thread_start_us":105,"threads_started":1}
I20260619 16:45:24.601509 18201 tablet_service.cc:1467] Processing CreateTablet for tablet 4929ab03e7174bd3bbcf8bfa11a1fc82 (DEFAULT_TABLE table=test-table [id=12984e927d0346efab6c80b7eb9779bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:45:24.601509 18070 tablet_service.cc:1467] Processing CreateTablet for tablet 4929ab03e7174bd3bbcf8bfa11a1fc82 (DEFAULT_TABLE table=test-table [id=12984e927d0346efab6c80b7eb9779bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:45:24.601501 18332 tablet_service.cc:1467] Processing CreateTablet for tablet 4929ab03e7174bd3bbcf8bfa11a1fc82 (DEFAULT_TABLE table=test-table [id=12984e927d0346efab6c80b7eb9779bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:45:24.602662 18332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4929ab03e7174bd3bbcf8bfa11a1fc82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:24.602667 18201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4929ab03e7174bd3bbcf8bfa11a1fc82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:24.602667 18070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4929ab03e7174bd3bbcf8bfa11a1fc82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:24.610983 18415 tablet_bootstrap.cc:492] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P d8ae0b57b4364e66b1e172bb5a859e60: Bootstrap starting.
I20260619 16:45:24.612071 18416 tablet_bootstrap.cc:492] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P 8b5d17e1cbf44f329b61e8eeb33e0bb4: Bootstrap starting.
I20260619 16:45:24.612135 18417 tablet_bootstrap.cc:492] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e: Bootstrap starting.
I20260619 16:45:24.614475 18415 tablet_bootstrap.cc:654] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P d8ae0b57b4364e66b1e172bb5a859e60: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:24.614507 18417 tablet_bootstrap.cc:654] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:24.614507 18416 tablet_bootstrap.cc:654] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P 8b5d17e1cbf44f329b61e8eeb33e0bb4: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:24.615351 18415 log.cc:824] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P d8ae0b57b4364e66b1e172bb5a859e60: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:24.615384 18416 log.cc:824] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P 8b5d17e1cbf44f329b61e8eeb33e0bb4: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:24.615403 18417 log.cc:824] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:24.617460 18416 tablet_bootstrap.cc:492] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P 8b5d17e1cbf44f329b61e8eeb33e0bb4: No bootstrap required, opened a new log
I20260619 16:45:24.617484 18417 tablet_bootstrap.cc:492] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e: No bootstrap required, opened a new log
I20260619 16:45:24.617712 18416 ts_tablet_manager.cc:1397] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P 8b5d17e1cbf44f329b61e8eeb33e0bb4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 16:45:24.617714 18417 ts_tablet_manager.cc:1397] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 16:45:24.620369 18415 tablet_bootstrap.cc:492] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P d8ae0b57b4364e66b1e172bb5a859e60: No bootstrap required, opened a new log
I20260619 16:45:24.620630 18415 ts_tablet_manager.cc:1397] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P d8ae0b57b4364e66b1e172bb5a859e60: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260619 16:45:24.621732 18416 raft_consensus.cc:348] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P 8b5d17e1cbf44f329b61e8eeb33e0bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ae0b57b4364e66b1e172bb5a859e60" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34461 } } peers { permanent_uuid: "c88595385b55432e9fb58f5ad52af35e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41323 } } peers { permanent_uuid: "8b5d17e1cbf44f329b61e8eeb33e0bb4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39741 } }
I20260619 16:45:24.622030 18416 raft_consensus.cc:374] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P 8b5d17e1cbf44f329b61e8eeb33e0bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:24.622085 18416 raft_consensus.cc:729] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P 8b5d17e1cbf44f329b61e8eeb33e0bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b5d17e1cbf44f329b61e8eeb33e0bb4, State: Initialized, Role: FOLLOWER
I20260619 16:45:24.622098 18417 raft_consensus.cc:348] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ae0b57b4364e66b1e172bb5a859e60" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34461 } } peers { permanent_uuid: "c88595385b55432e9fb58f5ad52af35e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41323 } } peers { permanent_uuid: "8b5d17e1cbf44f329b61e8eeb33e0bb4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39741 } }
I20260619 16:45:24.622361 18417 raft_consensus.cc:374] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:24.622423 18417 raft_consensus.cc:729] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c88595385b55432e9fb58f5ad52af35e, State: Initialized, Role: FOLLOWER
I20260619 16:45:24.622593 18416 consensus_queue.cc:260] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P 8b5d17e1cbf44f329b61e8eeb33e0bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ae0b57b4364e66b1e172bb5a859e60" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34461 } } peers { permanent_uuid: "c88595385b55432e9fb58f5ad52af35e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41323 } } peers { permanent_uuid: "8b5d17e1cbf44f329b61e8eeb33e0bb4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39741 } }
I20260619 16:45:24.622820 18417 consensus_queue.cc:260] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ae0b57b4364e66b1e172bb5a859e60" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34461 } } peers { permanent_uuid: "c88595385b55432e9fb58f5ad52af35e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41323 } } peers { permanent_uuid: "8b5d17e1cbf44f329b61e8eeb33e0bb4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39741 } }
I20260619 16:45:24.623549 18398 heartbeater.cc:503] Master 127.0.98.62:40133 was elected leader, sending a full tablet report...
I20260619 16:45:24.623821 18416 ts_tablet_manager.cc:1428] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P 8b5d17e1cbf44f329b61e8eeb33e0bb4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 16:45:24.623819 18417 ts_tablet_manager.cc:1428] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 16:45:24.624111 18136 heartbeater.cc:503] Master 127.0.98.62:40133 was elected leader, sending a full tablet report...
I20260619 16:45:24.624694 18415 raft_consensus.cc:348] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P d8ae0b57b4364e66b1e172bb5a859e60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ae0b57b4364e66b1e172bb5a859e60" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34461 } } peers { permanent_uuid: "c88595385b55432e9fb58f5ad52af35e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41323 } } peers { permanent_uuid: "8b5d17e1cbf44f329b61e8eeb33e0bb4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39741 } }
I20260619 16:45:24.625008 18415 raft_consensus.cc:374] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P d8ae0b57b4364e66b1e172bb5a859e60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:24.625062 18415 raft_consensus.cc:729] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P d8ae0b57b4364e66b1e172bb5a859e60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ae0b57b4364e66b1e172bb5a859e60, State: Initialized, Role: FOLLOWER
I20260619 16:45:24.625531 18415 consensus_queue.cc:260] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P d8ae0b57b4364e66b1e172bb5a859e60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ae0b57b4364e66b1e172bb5a859e60" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34461 } } peers { permanent_uuid: "c88595385b55432e9fb58f5ad52af35e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41323 } } peers { permanent_uuid: "8b5d17e1cbf44f329b61e8eeb33e0bb4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39741 } }
I20260619 16:45:24.626225 18267 heartbeater.cc:503] Master 127.0.98.62:40133 was elected leader, sending a full tablet report...
I20260619 16:45:24.626667 18415 ts_tablet_manager.cc:1428] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P d8ae0b57b4364e66b1e172bb5a859e60: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260619 16:45:24.628693 18137 tablet.cc:2369] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P 8b5d17e1cbf44f329b61e8eeb33e0bb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:45:24.741426 18422 raft_consensus.cc:482] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:45:24.741643 18422 raft_consensus.cc:504] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ae0b57b4364e66b1e172bb5a859e60" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34461 } } peers { permanent_uuid: "c88595385b55432e9fb58f5ad52af35e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41323 } } peers { permanent_uuid: "8b5d17e1cbf44f329b61e8eeb33e0bb4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39741 } }
I20260619 16:45:24.742805 18422 leader_election.cc:288] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ae0b57b4364e66b1e172bb5a859e60 (127.0.98.2:34461), 8b5d17e1cbf44f329b61e8eeb33e0bb4 (127.0.98.1:39741)
I20260619 16:45:24.747223 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:24.743153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39741 (local address 127.0.98.3:40105)
0619 16:45:24.743267 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:24.743315 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:24.743417 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:24.743715 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:24.743719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:24.743741 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:24.743878 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:24.743884 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.744906 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.744909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:24.745895 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:24.745904 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.746735 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.746739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:24.746842 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:24.746913 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 16:45:24.747001 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 16:45:24.747084 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 16:45:24.747272 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:24.743265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40105 (local address 127.0.98.1:39741)
0619 16:45:24.743408 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:24.743411 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:24.743426 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:45:24.743509 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:24.743512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:24.743566 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:24.743713 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:24.743998 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.744751 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.746025 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.746614 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.746692 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:24.746746 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:24.747044 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 16:45:24.747114 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:45:24.747517 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:24.743407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57085 (local address 127.0.98.2:34461)
0619 16:45:24.743548 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:24.743554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:24.744077 (+ 523us) server_negotiation.cc:407] Connection header received
0619 16:45:24.744176 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:24.744178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:24.744221 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:24.744299 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:24.744583 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.745268 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.746457 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.746956 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.747054 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:24.747108 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:24.747299 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 16:45:24.747388 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 16:45:24.747861 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:24.743153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34461 (local address 127.0.98.3:57085)
0619 16:45:24.743960 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:24.743976 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:24.744080 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:24.744337 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:24.744340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:24.744357 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:24.744472 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:24.744477 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.745388 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.745391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:24.746337 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:24.746347 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.747073 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.747077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:24.747171 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:24.747194 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:45:24.747271 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 16:45:24.747751 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":348,"thread_start_us":181,"threads_started":1}
I20260619 16:45:24.748044 18090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4929ab03e7174bd3bbcf8bfa11a1fc82" candidate_uuid: "c88595385b55432e9fb58f5ad52af35e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b5d17e1cbf44f329b61e8eeb33e0bb4" is_pre_election: true
I20260619 16:45:24.748396 18090 raft_consensus.cc:2454] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P 8b5d17e1cbf44f329b61e8eeb33e0bb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c88595385b55432e9fb58f5ad52af35e in term 0.
I20260619 16:45:24.748467 18221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4929ab03e7174bd3bbcf8bfa11a1fc82" candidate_uuid: "c88595385b55432e9fb58f5ad52af35e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ae0b57b4364e66b1e172bb5a859e60" is_pre_election: true
I20260619 16:45:24.748739 18221 raft_consensus.cc:2454] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P d8ae0b57b4364e66b1e172bb5a859e60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c88595385b55432e9fb58f5ad52af35e in term 0.
I20260619 16:45:24.749006 18288 leader_election.cc:302] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b5d17e1cbf44f329b61e8eeb33e0bb4, c88595385b55432e9fb58f5ad52af35e; no voters:
I20260619 16:45:24.749297 18422 raft_consensus.cc:2781] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:45:24.749397 18422 raft_consensus.cc:482] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:45:24.749454 18422 raft_consensus.cc:3037] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:24.750491 18422 raft_consensus.cc:504] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ae0b57b4364e66b1e172bb5a859e60" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34461 } } peers { permanent_uuid: "c88595385b55432e9fb58f5ad52af35e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41323 } } peers { permanent_uuid: "8b5d17e1cbf44f329b61e8eeb33e0bb4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39741 } }
I20260619 16:45:24.750811 18422 leader_election.cc:288] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e [CANDIDATE]: Term 1 election: Requested vote from peers d8ae0b57b4364e66b1e172bb5a859e60 (127.0.98.2:34461), 8b5d17e1cbf44f329b61e8eeb33e0bb4 (127.0.98.1:39741)
I20260619 16:45:24.751237 18090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4929ab03e7174bd3bbcf8bfa11a1fc82" candidate_uuid: "c88595385b55432e9fb58f5ad52af35e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b5d17e1cbf44f329b61e8eeb33e0bb4"
I20260619 16:45:24.751334 18221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4929ab03e7174bd3bbcf8bfa11a1fc82" candidate_uuid: "c88595385b55432e9fb58f5ad52af35e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ae0b57b4364e66b1e172bb5a859e60"
I20260619 16:45:24.751406 18090 raft_consensus.cc:3037] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P 8b5d17e1cbf44f329b61e8eeb33e0bb4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:24.751498 18221 raft_consensus.cc:3037] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P d8ae0b57b4364e66b1e172bb5a859e60 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:24.752555 18090 raft_consensus.cc:2454] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P 8b5d17e1cbf44f329b61e8eeb33e0bb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c88595385b55432e9fb58f5ad52af35e in term 1.
I20260619 16:45:24.752727 18221 raft_consensus.cc:2454] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P d8ae0b57b4364e66b1e172bb5a859e60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c88595385b55432e9fb58f5ad52af35e in term 1.
I20260619 16:45:24.752972 18288 leader_election.cc:302] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b5d17e1cbf44f329b61e8eeb33e0bb4, c88595385b55432e9fb58f5ad52af35e; no voters:
I20260619 16:45:24.753180 18422 raft_consensus.cc:2781] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:24.753417 18422 raft_consensus.cc:686] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e [term 1 LEADER]: Becoming Leader. State: Replica: c88595385b55432e9fb58f5ad52af35e, State: Running, Role: LEADER
I20260619 16:45:24.753764 18422 consensus_queue.cc:237] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ae0b57b4364e66b1e172bb5a859e60" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34461 } } peers { permanent_uuid: "c88595385b55432e9fb58f5ad52af35e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41323 } } peers { permanent_uuid: "8b5d17e1cbf44f329b61e8eeb33e0bb4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39741 } }
I20260619 16:45:24.757040 17947 catalog_manager.cc:5495] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e reported cstate change: term changed from 0 to 1, leader changed from <none> to c88595385b55432e9fb58f5ad52af35e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c88595385b55432e9fb58f5ad52af35e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ae0b57b4364e66b1e172bb5a859e60" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c88595385b55432e9fb58f5ad52af35e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b5d17e1cbf44f329b61e8eeb33e0bb4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39741 } health_report { overall_health: UNKNOWN } } }
I20260619 16:45:24.796869 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:24.792544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38726 (local address 127.0.98.3:41323)
0619 16:45:24.792676 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:24.792680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:24.792693 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:45:24.792937 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:24.792941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:24.793015 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:24.793124 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:45:24.793500 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.794017 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.795024 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:24.795250 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:24.795356 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:24.795798 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:45:24.796412 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:45:24.796566 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:24.796616 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 16:45:24.796676 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 16:45:24.797760 18399 tablet.cc:2369] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:45:24.804052 18221 raft_consensus.cc:1261] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P d8ae0b57b4364e66b1e172bb5a859e60 [term 1 FOLLOWER]: Refusing update from remote peer c88595385b55432e9fb58f5ad52af35e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:45:24.804052 18090 raft_consensus.cc:1261] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P 8b5d17e1cbf44f329b61e8eeb33e0bb4 [term 1 FOLLOWER]: Refusing update from remote peer c88595385b55432e9fb58f5ad52af35e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:45:24.805110 18425 consensus_queue.cc:1035] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ae0b57b4364e66b1e172bb5a859e60" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:45:24.805394 18422 consensus_queue.cc:1035] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b5d17e1cbf44f329b61e8eeb33e0bb4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:45:24.816394 18430 mvcc.cc:205] Tried to move back new op lower bound from 7298611301588455424 to 7298611301393960960. Current Snapshot: MvccSnapshot[applied={T|T < 7298611301588455424}]
I20260619 16:45:24.817538 18428 mvcc.cc:205] Tried to move back new op lower bound from 7298611301588455424 to 7298611301393960960. Current Snapshot: MvccSnapshot[applied={T|T < 7298611301588455424 or (T in {7298611301588455424})}]
I20260619 16:45:24.818982 18429 mvcc.cc:205] Tried to move back new op lower bound from 7298611301588455424 to 7298611301393960960. Current Snapshot: MvccSnapshot[applied={T|T < 7298611301588455424}]
I20260619 16:45:24.825778 17947 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:45:24.825974 17947 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:45:24.829833 17947 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db9396970c2b47e9ba481ed03f64ae79: Sending DeleteTablet for 3 replicas of tablet 4929ab03e7174bd3bbcf8bfa11a1fc82
I20260619 16:45:24.830901 18201 tablet_service.cc:1514] Processing DeleteTablet for tablet 4929ab03e7174bd3bbcf8bfa11a1fc82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:45:24 UTC) from {username='slave'} at 127.0.0.1:59404
I20260619 16:45:24.830900 18332 tablet_service.cc:1514] Processing DeleteTablet for tablet 4929ab03e7174bd3bbcf8bfa11a1fc82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:45:24 UTC) from {username='slave'} at 127.0.0.1:38714
I20260619 16:45:24.831547 18438 tablet_replica.cc:330] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P d8ae0b57b4364e66b1e172bb5a859e60: stopping tablet replica
I20260619 16:45:24.831508 18070 tablet_service.cc:1514] Processing DeleteTablet for tablet 4929ab03e7174bd3bbcf8bfa11a1fc82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:45:24 UTC) from {username='slave'} at 127.0.0.1:48206
I20260619 16:45:24.831977 18439 tablet_replica.cc:330] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P 8b5d17e1cbf44f329b61e8eeb33e0bb4: stopping tablet replica
I20260619 16:45:24.832202 18438 raft_consensus.cc:2229] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P d8ae0b57b4364e66b1e172bb5a859e60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:45:24.832259 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 18008
I20260619 16:45:24.832276 18439 raft_consensus.cc:2229] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P 8b5d17e1cbf44f329b61e8eeb33e0bb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:45:24.832691 18438 raft_consensus.cc:2258] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P d8ae0b57b4364e66b1e172bb5a859e60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:45:24.834653 18438 ts_tablet_manager.cc:1905] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P d8ae0b57b4364e66b1e172bb5a859e60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:45:24.837911 18438 ts_tablet_manager.cc:1918] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P d8ae0b57b4364e66b1e172bb5a859e60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:45:24.838055 18438 log.cc:1196] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P d8ae0b57b4364e66b1e172bb5a859e60: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-1/wal/wals/4929ab03e7174bd3bbcf8bfa11a1fc82
I20260619 16:45:24.838431 18438 ts_tablet_manager.cc:1939] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P d8ae0b57b4364e66b1e172bb5a859e60: Deleting consensus metadata
I20260619 16:45:24.840124 17932 catalog_manager.cc:4841] TS d8ae0b57b4364e66b1e172bb5a859e60 (127.0.98.2:34461): tablet 4929ab03e7174bd3bbcf8bfa11a1fc82 (table test-table [id=12984e927d0346efab6c80b7eb9779bb]) successfully deleted
W20260619 16:45:24.841384 17935 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39741 (error 108)
I20260619 16:45:24.842146 18437 tablet_replica.cc:330] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e: stopping tablet replica
I20260619 16:45:24.842538 18437 raft_consensus.cc:2229] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:45:24.842942 18437 raft_consensus.cc:2258] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 16:45:24.843514 18412 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:45:24.842498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39741 (local address 127.0.0.1:48208)
0619 16:45:24.842850 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:24.842869 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:24.842987 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:24.843398 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39741: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":158}
I20260619 16:45:24.843698 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 18139
W20260619 16:45:24.843741 17935 catalog_manager.cc:4584] TS 8b5d17e1cbf44f329b61e8eeb33e0bb4 (127.0.98.1:39741): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4929ab03e7174bd3bbcf8bfa11a1fc82: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39741: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 16:45:24.845288 18437 ts_tablet_manager.cc:1905] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:45:24.848994 18437 ts_tablet_manager.cc:1918] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:45:24.849148 18437 log.cc:1196] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887259574980-392-0/minicluster-data/ts-2/wal/wals/4929ab03e7174bd3bbcf8bfa11a1fc82
I20260619 16:45:24.849464 18437 ts_tablet_manager.cc:1939] T 4929ab03e7174bd3bbcf8bfa11a1fc82 P c88595385b55432e9fb58f5ad52af35e: Deleting consensus metadata
I20260619 16:45:24.850744 17933 catalog_manager.cc:4841] TS c88595385b55432e9fb58f5ad52af35e (127.0.98.3:41323): tablet 4929ab03e7174bd3bbcf8bfa11a1fc82 (table test-table [id=12984e927d0346efab6c80b7eb9779bb]) successfully deleted
I20260619 16:45:24.851727 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 18270
I20260619 16:45:24.857734 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 17917
2026-06-19T16:45:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1285 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:45:24 dist-test-slave-03cd krb5kdc[18443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:45:24 dist-test-slave-03cd krb5kdc[18443](info): set up 2 sockets
Jun 19 16:45:24 dist-test-slave-03cd krb5kdc[18443](info): commencing operation
krb5kdc: starting...
W20260619 16:45:25.927742 392 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:45:26 dist-test-slave-03cd krb5kdc[18443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:45:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:45:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:45:26.064321 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40613
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:26.186241 18456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:26.186607 18456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:26.186686 18456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:26.191108 18456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:45:26.191246 18456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:26.191293 18456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:26.191325 18456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:45:26.191365 18456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:45:26.197712 18456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:26.199158 18456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:26.200340 18456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:26.209859 18461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:26.209898 18462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:26.209908 18464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:26.210135 18456 server_base.cc:1034] running on GCE node
I20260619 16:45:26.210919 18456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:26.212193 18456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:26.213426 18456 hybrid_clock.cc:648] HybridClock initialized: now 1781887526213407 us; error 32 us; skew 500 ppm
Jun 19 16:45:26 dist-test-slave-03cd krb5kdc[18443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:26.222905 18456 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:45:26.224176 18456 webserver.cc:460] Webserver started at http://127.0.98.62:38457/ using document root <none> and password file <none>
I20260619 16:45:26.224985 18456 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:26.225111 18456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:26.225383 18456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:26.227739 18456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "4df3efe8e8764e25a747bbe2cf4974f5"
format_stamp: "Formatted at 2026-06-19 16:45:26 on dist-test-slave-03cd"
server_key: "7cf2a78d85bbd94f4e5ace9420737bb7"
server_key_iv: "244cf405a0f86531482eeeb14668b824"
server_key_version: "encryptionkey@0"
I20260619 16:45:26.228391 18456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "4df3efe8e8764e25a747bbe2cf4974f5"
format_stamp: "Formatted at 2026-06-19 16:45:26 on dist-test-slave-03cd"
server_key: "7cf2a78d85bbd94f4e5ace9420737bb7"
server_key_iv: "244cf405a0f86531482eeeb14668b824"
server_key_version: "encryptionkey@0"
I20260619 16:45:26.232790 18456 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:45:26.235617 18471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:26.237179 18456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:45:26.237376 18456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "4df3efe8e8764e25a747bbe2cf4974f5"
format_stamp: "Formatted at 2026-06-19 16:45:26 on dist-test-slave-03cd"
server_key: "7cf2a78d85bbd94f4e5ace9420737bb7"
server_key_iv: "244cf405a0f86531482eeeb14668b824"
server_key_version: "encryptionkey@0"
I20260619 16:45:26.237543 18456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:26.250376 18456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:26.254122 18456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:26.254377 18456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:26.262306 18456 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40613
I20260619 16:45:26.262352 18523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40613 every 8 connection(s)
I20260619 16:45:26.263337 18456 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:45:26.266275 18524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:26.272117 18524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4df3efe8e8764e25a747bbe2cf4974f5: Bootstrap starting.
I20260619 16:45:26.272490 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 18456
I20260619 16:45:26.272624 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:45:26.272922 392 external_mini_cluster.cc:1428] Setting key 56d88da7af91f3656470e4be0a59519d
I20260619 16:45:26.274946 18524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4df3efe8e8764e25a747bbe2cf4974f5: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:26.275761 18524 log.cc:824] T 00000000000000000000000000000000 P 4df3efe8e8764e25a747bbe2cf4974f5: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:26.278167 18524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4df3efe8e8764e25a747bbe2cf4974f5: No bootstrap required, opened a new log
Jun 19 16:45:26 dist-test-slave-03cd krb5kdc[18443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:45:26.282874 18524 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4df3efe8e8764e25a747bbe2cf4974f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df3efe8e8764e25a747bbe2cf4974f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40613 } }
I20260619 16:45:26.283232 18524 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4df3efe8e8764e25a747bbe2cf4974f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:26.283329 18524 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4df3efe8e8764e25a747bbe2cf4974f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df3efe8e8764e25a747bbe2cf4974f5, State: Initialized, Role: FOLLOWER
I20260619 16:45:26.283891 18524 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4df3efe8e8764e25a747bbe2cf4974f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df3efe8e8764e25a747bbe2cf4974f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40613 } }
I20260619 16:45:26.284075 18524 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4df3efe8e8764e25a747bbe2cf4974f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:45:26.284144 18524 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4df3efe8e8764e25a747bbe2cf4974f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:45:26.284242 18524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4df3efe8e8764e25a747bbe2cf4974f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:26.285333 18524 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4df3efe8e8764e25a747bbe2cf4974f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df3efe8e8764e25a747bbe2cf4974f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40613 } }
I20260619 16:45:26.285775 18524 leader_election.cc:302] T 00000000000000000000000000000000 P 4df3efe8e8764e25a747bbe2cf4974f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4df3efe8e8764e25a747bbe2cf4974f5; no voters:
I20260619 16:45:26.286273 18524 leader_election.cc:288] T 00000000000000000000000000000000 P 4df3efe8e8764e25a747bbe2cf4974f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:45:26.286432 18529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4df3efe8e8764e25a747bbe2cf4974f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:26.286639 18529 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4df3efe8e8764e25a747bbe2cf4974f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4df3efe8e8764e25a747bbe2cf4974f5, State: Running, Role: LEADER
I20260619 16:45:26.286980 18529 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4df3efe8e8764e25a747bbe2cf4974f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df3efe8e8764e25a747bbe2cf4974f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40613 } }
I20260619 16:45:26.287990 18524 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4df3efe8e8764e25a747bbe2cf4974f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:45:26.288969 18530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4df3efe8e8764e25a747bbe2cf4974f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4df3efe8e8764e25a747bbe2cf4974f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df3efe8e8764e25a747bbe2cf4974f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40613 } } }
I20260619 16:45:26.289155 18530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4df3efe8e8764e25a747bbe2cf4974f5 [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:26.289482 18531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4df3efe8e8764e25a747bbe2cf4974f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4df3efe8e8764e25a747bbe2cf4974f5. Latest consensus state: current_term: 1 leader_uuid: "4df3efe8e8764e25a747bbe2cf4974f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df3efe8e8764e25a747bbe2cf4974f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40613 } } }
I20260619 16:45:26.289585 18531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4df3efe8e8764e25a747bbe2cf4974f5 [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:26.289882 18538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:45:26.290067 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:26.274384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59770 (local address 127.0.98.62:40613)
0619 16:45:26.274842 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:26.274861 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:26.274899 (+ 38us) server_negotiation.cc:407] Connection header received
0619 16:45:26.275729 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:26.275764 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:26.276174 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:26.276491 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:26.277237 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:26.278827 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:26.279835 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:26.280684 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:26.280989 (+ 305us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:26.283511 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:26.283518 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:26.283535 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:26.283574 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:26.285741 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:26.286281 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:26.286287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:26.286293 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:26.286478 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:26.286877 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:26.286879 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:26.286881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:26.287398 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:26.287626 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:26.288366 (+ 740us) server_negotiation.cc:299] Negotiation successful
0619 16:45:26.288694 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":162,"threads_started":1}
I20260619 16:45:26.293933 18538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:45:26.300819 18538 catalog_manager.cc:1347] Generated new cluster ID: bfc86e09a24b4d548bdb660f38866cee
I20260619 16:45:26.300909 18538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:45:26.307857 18538 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:45:26.308918 18538 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:45:26.314337 18538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4df3efe8e8764e25a747bbe2cf4974f5: Generated new TSK 0
I20260619 16:45:26.315222 18538 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:45:26.356633 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40613
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:26.479737 18550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:26.480103 18550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:26.480182 18550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:26.484540 18550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:26.484644 18550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:26.484738 18550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:45:26.490833 18550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:26.492118 18550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:26.493180 18550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:26.503291 18558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:26.503270 18555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:26.503270 18556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:26.504110 18550 server_base.cc:1034] running on GCE node
I20260619 16:45:26.504496 18550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:26.505162 18550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:26.506351 18550 hybrid_clock.cc:648] HybridClock initialized: now 1781887526506324 us; error 42 us; skew 500 ppm
Jun 19 16:45:26 dist-test-slave-03cd krb5kdc[18443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:26.509644 18550 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:45:26.510694 18550 webserver.cc:460] Webserver started at http://127.0.98.1:33667/ using document root <none> and password file <none>
I20260619 16:45:26.511315 18550 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:26.511415 18550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:26.511649 18550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:26.514019 18550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-0/data/instance:
uuid: "f52337e2cb7f4a8d9151f6ee80fabb8e"
format_stamp: "Formatted at 2026-06-19 16:45:26 on dist-test-slave-03cd"
server_key: "27977f9940e3865f531ced1ec4e382f7"
server_key_iv: "6e03827b47b55d547ce2b62c8199f4c2"
server_key_version: "encryptionkey@0"
I20260619 16:45:26.514647 18550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f52337e2cb7f4a8d9151f6ee80fabb8e"
format_stamp: "Formatted at 2026-06-19 16:45:26 on dist-test-slave-03cd"
server_key: "27977f9940e3865f531ced1ec4e382f7"
server_key_iv: "6e03827b47b55d547ce2b62c8199f4c2"
server_key_version: "encryptionkey@0"
I20260619 16:45:26.518839 18550 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260619 16:45:26.521550 18565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:26.522944 18550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:45:26.523108 18550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "f52337e2cb7f4a8d9151f6ee80fabb8e"
format_stamp: "Formatted at 2026-06-19 16:45:26 on dist-test-slave-03cd"
server_key: "27977f9940e3865f531ced1ec4e382f7"
server_key_iv: "6e03827b47b55d547ce2b62c8199f4c2"
server_key_version: "encryptionkey@0"
I20260619 16:45:26.523241 18550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:26.542631 18550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:26.546409 18550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:26.546643 18550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:26.547277 18550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:45:26.548193 18550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:45:26.548243 18550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:26.548326 18550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:45:26.548357 18550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:26.557224 18550 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40165
I20260619 16:45:26.557327 18678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40165 every 8 connection(s)
I20260619 16:45:26.558234 18550 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:45:26.564508 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 18550
I20260619 16:45:26.564661 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:45:26.564960 392 external_mini_cluster.cc:1428] Setting key 0dbd55b36ac9ac757936c734eec9a8dd
Jun 19 16:45:26 dist-test-slave-03cd krb5kdc[18443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:45:26.575112 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:26.560324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40757 (local address 127.0.98.62:40613)
0619 16:45:26.560477 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:26.560481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:26.561391 (+ 910us) server_negotiation.cc:407] Connection header received
0619 16:45:26.562748 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:26.562751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:26.562805 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:26.562885 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:26.564329 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:26.565134 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:26.566634 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:26.567096 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:26.567283 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:26.570481 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:26.570489 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:26.570495 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:26.570535 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:26.572460 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:26.573094 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:26.573101 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:26.573106 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:26.573216 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:26.573693 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:26.573699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:26.573701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:26.573977 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:26.574099 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:26.574729 (+ 630us) server_negotiation.cc:299] Negotiation successful
0619 16:45:26.574888 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:45:26.576057 18681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:26.560720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40613 (local address 127.0.98.1:40757)
0619 16:45:26.561219 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:26.561265 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:26.562518 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:26.563097 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:26.563104 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:26.563507 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:45:26.564124 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:26.564139 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:26.565299 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:26.565302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:26.566515 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:26.566526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:26.567319 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:26.567322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:26.567500 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:26.568287 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:45:26.568304 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:45:26.570253 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:45:26.572637 (+ 2384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:45:26.572642 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:45:26.572660 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:45:26.572978 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:45:26.573381 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:45:26.573384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:45:26.573385 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:45:26.573574 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:45:26.574132 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:45:26.574139 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:45:26.574452 (+ 313us) client_negotiation.cc:769] Sending connection context
0619 16:45:26.574646 (+ 194us) client_negotiation.cc:240] Negotiation successful
0619 16:45:26.574949 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":127,"threads_started":1}
I20260619 16:45:26.577611 18679 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40613
I20260619 16:45:26.577853 18679 heartbeater.cc:464] Registering TS with master...
I20260619 16:45:26.578416 18679 heartbeater.cc:511] Master 127.0.98.62:40613 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 16:45:26.580204 18488 ts_manager.cc:194] Registered new tserver with Master: f52337e2cb7f4a8d9151f6ee80fabb8e (127.0.98.1:40165)
I20260619 16:45:26.581459 18488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:45:26.595312 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40613
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:26.713954 18684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:26.714279 18684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:26.714334 18684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:26.718888 18684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:26.718984 18684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:26.719050 18684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:45:26.725523 18684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:26.726838 18684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:26.727993 18684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:26.738515 18689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:26.738554 18692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:26.738570 18690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:26.738560 18684 server_base.cc:1034] running on GCE node
I20260619 16:45:26.739225 18684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:26.740034 18684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:26.741220 18684 hybrid_clock.cc:648] HybridClock initialized: now 1781887526741202 us; error 36 us; skew 500 ppm
Jun 19 16:45:26 dist-test-slave-03cd krb5kdc[18443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:26.744444 18684 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:45:26.745461 18684 webserver.cc:460] Webserver started at http://127.0.98.2:42039/ using document root <none> and password file <none>
I20260619 16:45:26.746068 18684 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:26.746160 18684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:26.746397 18684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:26.748585 18684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-1/data/instance:
uuid: "2b207254652442118fe0a63878865cde"
format_stamp: "Formatted at 2026-06-19 16:45:26 on dist-test-slave-03cd"
server_key: "60f448c335bdf5121bf4ea8fc9cce64d"
server_key_iv: "52bf8aa24669654e4610dc7114ccb600"
server_key_version: "encryptionkey@0"
I20260619 16:45:26.749172 18684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2b207254652442118fe0a63878865cde"
format_stamp: "Formatted at 2026-06-19 16:45:26 on dist-test-slave-03cd"
server_key: "60f448c335bdf5121bf4ea8fc9cce64d"
server_key_iv: "52bf8aa24669654e4610dc7114ccb600"
server_key_version: "encryptionkey@0"
I20260619 16:45:26.753155 18684 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:45:26.755904 18699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:26.757256 18684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:45:26.757431 18684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "2b207254652442118fe0a63878865cde"
format_stamp: "Formatted at 2026-06-19 16:45:26 on dist-test-slave-03cd"
server_key: "60f448c335bdf5121bf4ea8fc9cce64d"
server_key_iv: "52bf8aa24669654e4610dc7114ccb600"
server_key_version: "encryptionkey@0"
I20260619 16:45:26.757570 18684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:26.767151 18684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:26.770702 18684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:26.770959 18684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:26.771623 18684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:45:26.772572 18684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:45:26.772645 18684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:26.772718 18684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:45:26.772763 18684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:26.782411 18684 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44279
I20260619 16:45:26.782433 18812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44279 every 8 connection(s)
I20260619 16:45:26.783375 18684 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:45:26.792512 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 18684
I20260619 16:45:26.792651 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:45:26.792973 392 external_mini_cluster.cc:1428] Setting key 4ade62e91f97df3831dec0a5e3e6cc67
Jun 19 16:45:26 dist-test-slave-03cd krb5kdc[18443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:45:26.799353 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:26.785340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44771 (local address 127.0.98.62:40613)
0619 16:45:26.785525 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:26.785529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:26.786381 (+ 852us) server_negotiation.cc:407] Connection header received
0619 16:45:26.787820 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:26.787824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:26.787888 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:26.788005 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:26.789517 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:26.790078 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:26.791326 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:26.791573 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:26.791719 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:26.794988 (+ 3269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:26.794992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:26.794995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:26.795033 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:26.796730 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:26.797411 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:26.797415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:26.797416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:26.797552 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:26.798000 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:26.798003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:26.798004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:26.798215 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:26.798307 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:26.799075 (+ 768us) server_negotiation.cc:299] Negotiation successful
0619 16:45:26.799203 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 16:45:26.800731 18815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:26.785732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40613 (local address 127.0.98.2:44771)
0619 16:45:26.786208 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:26.786252 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:26.787530 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:26.788242 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:26.788249 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:26.788656 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:45:26.789343 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:26.789360 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:26.790213 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:26.790216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:26.791182 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:26.791192 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:26.791776 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:26.791780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:26.791980 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:26.792633 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:45:26.792656 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:45:26.794761 (+ 2105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:45:26.796891 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:45:26.796897 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:45:26.796915 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:45:26.797274 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:45:26.797683 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:45:26.797686 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:45:26.797689 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:45:26.797872 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:45:26.798348 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:45:26.798358 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:45:26.798742 (+ 384us) client_negotiation.cc:769] Sending connection context
0619 16:45:26.798986 (+ 244us) client_negotiation.cc:240] Negotiation successful
0619 16:45:26.799426 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":124,"threads_started":1}
I20260619 16:45:26.802277 18813 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40613
I20260619 16:45:26.802557 18813 heartbeater.cc:464] Registering TS with master...
I20260619 16:45:26.803092 18813 heartbeater.cc:511] Master 127.0.98.62:40613 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:45:26.804246 18488 ts_manager.cc:194] Registered new tserver with Master: 2b207254652442118fe0a63878865cde (127.0.98.2:44279)
I20260619 16:45:26.804913 18488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:45:26.823612 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40613
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:26.939206 18818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:26.939502 18818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:26.939546 18818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:26.943923 18818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:26.944000 18818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:26.944061 18818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:45:26.949725 18818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:26.950902 18818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:26.951926 18818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:26.962821 18826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:26.963039 18823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:26.963034 18818 server_base.cc:1034] running on GCE node
W20260619 16:45:26.963179 18824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:26.963604 18818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:26.964413 18818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:26.965612 18818 hybrid_clock.cc:648] HybridClock initialized: now 1781887526965597 us; error 34 us; skew 500 ppm
Jun 19 16:45:26 dist-test-slave-03cd krb5kdc[18443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:26.968660 18818 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:45:26.969630 18818 webserver.cc:460] Webserver started at http://127.0.98.3:43677/ using document root <none> and password file <none>
I20260619 16:45:26.970227 18818 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:26.970335 18818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:26.970572 18818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:26.972749 18818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-2/data/instance:
uuid: "2aec0408e10740c69d2cd490ee3bbc68"
format_stamp: "Formatted at 2026-06-19 16:45:26 on dist-test-slave-03cd"
server_key: "24af2e5aa3687002340c2a953d1ee698"
server_key_iv: "f76c204fa3d7a3915a42f526045de57d"
server_key_version: "encryptionkey@0"
I20260619 16:45:26.973330 18818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2aec0408e10740c69d2cd490ee3bbc68"
format_stamp: "Formatted at 2026-06-19 16:45:26 on dist-test-slave-03cd"
server_key: "24af2e5aa3687002340c2a953d1ee698"
server_key_iv: "f76c204fa3d7a3915a42f526045de57d"
server_key_version: "encryptionkey@0"
I20260619 16:45:26.977324 18818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:45:26.980074 18833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:26.981390 18818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:45:26.981540 18818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "2aec0408e10740c69d2cd490ee3bbc68"
format_stamp: "Formatted at 2026-06-19 16:45:26 on dist-test-slave-03cd"
server_key: "24af2e5aa3687002340c2a953d1ee698"
server_key_iv: "f76c204fa3d7a3915a42f526045de57d"
server_key_version: "encryptionkey@0"
I20260619 16:45:26.981667 18818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:27.006271 18818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:27.010351 18818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:27.010627 18818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:27.011475 18818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:45:27.012471 18818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:45:27.012521 18818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:27.012566 18818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:45:27.012583 18818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:27.022083 18818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34703
I20260619 16:45:27.022149 18946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34703 every 8 connection(s)
I20260619 16:45:27.023092 18818 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:45:27.031723 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 18818
I20260619 16:45:27.031919 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:45:27.032274 392 external_mini_cluster.cc:1428] Setting key 0e85047089425a281e2600bf1734ccb2
Jun 19 16:45:27 dist-test-slave-03cd krb5kdc[18443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:45:27.038417 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:27.025084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34071 (local address 127.0.98.62:40613)
0619 16:45:27.025247 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:27.025251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:27.026198 (+ 947us) server_negotiation.cc:407] Connection header received
0619 16:45:27.027728 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:27.027731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:27.027801 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:27.027912 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:27.029214 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:27.029766 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:27.030980 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:27.031205 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:27.031322 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:27.034489 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:27.034493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:27.034496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:27.034534 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:27.036075 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:27.036650 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:27.036655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:27.036657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:27.036745 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:27.037194 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:27.037198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:27.037200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:27.037429 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:27.037538 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:27.038113 (+ 575us) server_negotiation.cc:299] Negotiation successful
0619 16:45:27.038246 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:45:27.039378 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:27.025521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40613 (local address 127.0.98.3:34071)
0619 16:45:27.026019 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:27.026080 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:27.027430 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:27.028108 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:27.028116 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:27.028519 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:45:27.029043 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:27.029057 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:27.029920 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:27.029923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:27.030837 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:27.030847 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:27.031372 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:27.031376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:27.031555 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:27.032329 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:45:27.032356 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:45:27.034247 (+ 1891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:45:27.036211 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:45:27.036218 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:45:27.036236 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:45:27.036536 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:45:27.036910 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:45:27.036914 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:45:27.036916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:45:27.037083 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:45:27.037552 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:45:27.037561 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:45:27.037853 (+ 292us) client_negotiation.cc:769] Sending connection context
0619 16:45:27.038068 (+ 215us) client_negotiation.cc:240] Negotiation successful
0619 16:45:27.038468 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":128,"threads_started":1}
I20260619 16:45:27.040663 18947 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40613
I20260619 16:45:27.040934 18947 heartbeater.cc:464] Registering TS with master...
I20260619 16:45:27.041461 18947 heartbeater.cc:511] Master 127.0.98.62:40613 requested a full tablet report, sending...
I20260619 16:45:27.042582 18488 ts_manager.cc:194] Registered new tserver with Master: 2aec0408e10740c69d2cd490ee3bbc68 (127.0.98.3:34703)
I20260619 16:45:27.043186 18488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34071
I20260619 16:45:27.046459 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:45:27.056742 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:27.048428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59778 (local address 127.0.98.62:40613)
0619 16:45:27.048556 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:27.048561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:27.048775 (+ 214us) server_negotiation.cc:407] Connection header received
0619 16:45:27.048864 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:27.048866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:27.048908 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:27.048973 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:27.049690 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:27.050234 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:27.051139 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:27.051366 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:27.051475 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:27.053200 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:27.053203 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:27.053206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:27.053232 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:27.054710 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:27.055229 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:27.055232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:27.055234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:27.055336 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:27.055771 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:27.055774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:27.055775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:27.056007 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:27.056131 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:27.056479 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 16:45:27.056596 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:45:27.060526 18488 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:45:27.064247 18488 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:45:27.080745 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:27.075275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43320 (local address 127.0.98.1:40165)
0619 16:45:27.075647 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:27.075695 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:27.075748 (+ 53us) server_negotiation.cc:407] Connection header received
0619 16:45:27.076005 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:27.076012 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:27.076245 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:27.076415 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:27.076995 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:27.077780 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:27.079329 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:27.080007 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:27.080135 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:27.080297 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:27.080480 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 16:45:27.080579 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":72,"threads_started":1}
I20260619 16:45:27.080737 18961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:27.074455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40165 (local address 127.0.0.1:43320)
0619 16:45:27.075638 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:27.075652 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:27.075794 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:27.076413 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:27.076419 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:27.076615 (+ 196us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:27.076859 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:27.076864 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:27.077911 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:27.077916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:27.079187 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:27.079197 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:27.080137 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:27.080140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:27.080248 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:27.080296 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 16:45:27.080447 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 16:45:27.080578 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1127,"mutex_wait_us":31,"spinlock_wait_cycles":2304,"thread_start_us":134,"threads_started":1}
I20260619 16:45:27.082116 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:27.074754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34703 (local address 127.0.0.1:40254)
0619 16:45:27.076126 (+ 1372us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:27.076140 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:27.076229 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:27.076695 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:27.076698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:27.076712 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:27.076855 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:27.076863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:27.078591 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:27.078595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:27.080202 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:27.080209 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:27.081670 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:27.081673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:27.081781 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:27.081804 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:45:27.081892 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 16:45:27.081969 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1317,"mutex_wait_us":303,"thread_start_us":105,"threads_started":1}
I20260619 16:45:27.082712 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:27.075045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40254 (local address 127.0.98.3:34703)
0619 16:45:27.075992 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:27.076004 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:27.076250 (+ 246us) server_negotiation.cc:407] Connection header received
0619 16:45:27.076344 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:27.076352 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:27.076545 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:27.076707 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:27.077052 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:27.078405 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:27.080354 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:27.081347 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:27.081437 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:27.081565 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:27.082324 (+ 759us) server_negotiation.cc:299] Negotiation successful
0619 16:45:27.082547 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":853,"thread_start_us":97,"threads_started":1}
I20260619 16:45:27.083185 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:27.074386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44279 (local address 127.0.0.1:47118)
0619 16:45:27.075480 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:27.075545 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:27.075808 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:27.076606 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:27.076613 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:27.076638 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:27.076838 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:27.076849 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:27.078591 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:27.078595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:27.080179 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:27.080191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:27.082708 (+ 2517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:27.082714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:27.082828 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:27.082854 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 16:45:27.082937 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 16:45:27.083026 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"mutex_wait_us":68,"spinlock_wait_cycles":2944,"thread_start_us":90,"threads_started":1}
I20260619 16:45:27.083603 18960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:27.074546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47118 (local address 127.0.98.2:44279)
0619 16:45:27.075503 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:27.075517 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:27.075756 (+ 239us) server_negotiation.cc:407] Connection header received
0619 16:45:27.076007 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:27.076013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:27.076254 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:27.076417 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:27.076958 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:27.078427 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:27.080347 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:27.082258 (+ 1911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:27.082367 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:27.082633 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:27.083403 (+ 770us) server_negotiation.cc:299] Negotiation successful
0619 16:45:27.083491 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":847,"thread_start_us":137,"threads_started":1}
I20260619 16:45:27.084460 18613 tablet_service.cc:1467] Processing CreateTablet for tablet cee18cda2d98407c81c563bd5239dbac (DEFAULT_TABLE table=test-table [id=5ec38d387a79400c9cfc68033f8c7098]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:45:27.085500 18613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cee18cda2d98407c81c563bd5239dbac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:27.088593 18747 tablet_service.cc:1467] Processing CreateTablet for tablet cee18cda2d98407c81c563bd5239dbac (DEFAULT_TABLE table=test-table [id=5ec38d387a79400c9cfc68033f8c7098]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:45:27.088809 18881 tablet_service.cc:1467] Processing CreateTablet for tablet cee18cda2d98407c81c563bd5239dbac (DEFAULT_TABLE table=test-table [id=5ec38d387a79400c9cfc68033f8c7098]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:45:27.089737 18747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cee18cda2d98407c81c563bd5239dbac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:27.089851 18881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cee18cda2d98407c81c563bd5239dbac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:27.094485 18964 tablet_bootstrap.cc:492] T cee18cda2d98407c81c563bd5239dbac P f52337e2cb7f4a8d9151f6ee80fabb8e: Bootstrap starting.
I20260619 16:45:27.097141 18964 tablet_bootstrap.cc:654] T cee18cda2d98407c81c563bd5239dbac P f52337e2cb7f4a8d9151f6ee80fabb8e: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:27.097141 18965 tablet_bootstrap.cc:492] T cee18cda2d98407c81c563bd5239dbac P 2b207254652442118fe0a63878865cde: Bootstrap starting.
I20260619 16:45:27.098878 18966 tablet_bootstrap.cc:492] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68: Bootstrap starting.
I20260619 16:45:27.098902 18964 log.cc:824] T cee18cda2d98407c81c563bd5239dbac P f52337e2cb7f4a8d9151f6ee80fabb8e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:27.100383 18965 tablet_bootstrap.cc:654] T cee18cda2d98407c81c563bd5239dbac P 2b207254652442118fe0a63878865cde: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:27.101292 18966 tablet_bootstrap.cc:654] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:27.101285 18965 log.cc:824] T cee18cda2d98407c81c563bd5239dbac P 2b207254652442118fe0a63878865cde: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:27.101336 18964 tablet_bootstrap.cc:492] T cee18cda2d98407c81c563bd5239dbac P f52337e2cb7f4a8d9151f6ee80fabb8e: No bootstrap required, opened a new log
I20260619 16:45:27.101577 18964 ts_tablet_manager.cc:1397] T cee18cda2d98407c81c563bd5239dbac P f52337e2cb7f4a8d9151f6ee80fabb8e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 16:45:27.102702 18966 log.cc:824] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:27.103581 18965 tablet_bootstrap.cc:492] T cee18cda2d98407c81c563bd5239dbac P 2b207254652442118fe0a63878865cde: No bootstrap required, opened a new log
I20260619 16:45:27.103863 18965 ts_tablet_manager.cc:1397] T cee18cda2d98407c81c563bd5239dbac P 2b207254652442118fe0a63878865cde: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.002s
I20260619 16:45:27.104934 18966 tablet_bootstrap.cc:492] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68: No bootstrap required, opened a new log
I20260619 16:45:27.105129 18966 ts_tablet_manager.cc:1397] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 16:45:27.105965 18964 raft_consensus.cc:348] T cee18cda2d98407c81c563bd5239dbac P f52337e2cb7f4a8d9151f6ee80fabb8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b207254652442118fe0a63878865cde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } } peers { permanent_uuid: "f52337e2cb7f4a8d9151f6ee80fabb8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40165 } } peers { permanent_uuid: "2aec0408e10740c69d2cd490ee3bbc68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34703 } }
I20260619 16:45:27.106276 18964 raft_consensus.cc:374] T cee18cda2d98407c81c563bd5239dbac P f52337e2cb7f4a8d9151f6ee80fabb8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:27.106345 18964 raft_consensus.cc:729] T cee18cda2d98407c81c563bd5239dbac P f52337e2cb7f4a8d9151f6ee80fabb8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f52337e2cb7f4a8d9151f6ee80fabb8e, State: Initialized, Role: FOLLOWER
I20260619 16:45:27.106878 18964 consensus_queue.cc:260] T cee18cda2d98407c81c563bd5239dbac P f52337e2cb7f4a8d9151f6ee80fabb8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b207254652442118fe0a63878865cde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } } peers { permanent_uuid: "f52337e2cb7f4a8d9151f6ee80fabb8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40165 } } peers { permanent_uuid: "2aec0408e10740c69d2cd490ee3bbc68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34703 } }
I20260619 16:45:27.107651 18679 heartbeater.cc:503] Master 127.0.98.62:40613 was elected leader, sending a full tablet report...
I20260619 16:45:27.108047 18964 ts_tablet_manager.cc:1428] T cee18cda2d98407c81c563bd5239dbac P f52337e2cb7f4a8d9151f6ee80fabb8e: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260619 16:45:27.108601 18965 raft_consensus.cc:348] T cee18cda2d98407c81c563bd5239dbac P 2b207254652442118fe0a63878865cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b207254652442118fe0a63878865cde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } } peers { permanent_uuid: "f52337e2cb7f4a8d9151f6ee80fabb8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40165 } } peers { permanent_uuid: "2aec0408e10740c69d2cd490ee3bbc68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34703 } }
I20260619 16:45:27.108911 18965 raft_consensus.cc:374] T cee18cda2d98407c81c563bd5239dbac P 2b207254652442118fe0a63878865cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:27.108973 18965 raft_consensus.cc:729] T cee18cda2d98407c81c563bd5239dbac P 2b207254652442118fe0a63878865cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b207254652442118fe0a63878865cde, State: Initialized, Role: FOLLOWER
I20260619 16:45:27.108983 18966 raft_consensus.cc:348] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b207254652442118fe0a63878865cde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } } peers { permanent_uuid: "f52337e2cb7f4a8d9151f6ee80fabb8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40165 } } peers { permanent_uuid: "2aec0408e10740c69d2cd490ee3bbc68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34703 } }
I20260619 16:45:27.109246 18966 raft_consensus.cc:374] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:27.109293 18966 raft_consensus.cc:729] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aec0408e10740c69d2cd490ee3bbc68, State: Initialized, Role: FOLLOWER
I20260619 16:45:27.109413 18965 consensus_queue.cc:260] T cee18cda2d98407c81c563bd5239dbac P 2b207254652442118fe0a63878865cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b207254652442118fe0a63878865cde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } } peers { permanent_uuid: "f52337e2cb7f4a8d9151f6ee80fabb8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40165 } } peers { permanent_uuid: "2aec0408e10740c69d2cd490ee3bbc68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34703 } }
I20260619 16:45:27.109663 18966 consensus_queue.cc:260] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b207254652442118fe0a63878865cde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } } peers { permanent_uuid: "f52337e2cb7f4a8d9151f6ee80fabb8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40165 } } peers { permanent_uuid: "2aec0408e10740c69d2cd490ee3bbc68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34703 } }
I20260619 16:45:27.110110 18813 heartbeater.cc:503] Master 127.0.98.62:40613 was elected leader, sending a full tablet report...
I20260619 16:45:27.110641 18966 ts_tablet_manager.cc:1428] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 16:45:27.110958 18947 heartbeater.cc:503] Master 127.0.98.62:40613 was elected leader, sending a full tablet report...
I20260619 16:45:27.111164 18965 ts_tablet_manager.cc:1428] T cee18cda2d98407c81c563bd5239dbac P 2b207254652442118fe0a63878865cde: Time spent starting tablet: real 0.007s user 0.001s sys 0.004s
W20260619 16:45:27.274142 18948 tablet.cc:2369] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:45:27.284673 18814 tablet.cc:2369] T cee18cda2d98407c81c563bd5239dbac P 2b207254652442118fe0a63878865cde: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:45:27.309767 18680 tablet.cc:2369] T cee18cda2d98407c81c563bd5239dbac P f52337e2cb7f4a8d9151f6ee80fabb8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:45:27.490481 18972 raft_consensus.cc:482] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:45:27.490756 18972 raft_consensus.cc:504] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b207254652442118fe0a63878865cde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } } peers { permanent_uuid: "f52337e2cb7f4a8d9151f6ee80fabb8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40165 } } peers { permanent_uuid: "2aec0408e10740c69d2cd490ee3bbc68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34703 } }
I20260619 16:45:27.492431 18972 leader_election.cc:288] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b207254652442118fe0a63878865cde (127.0.98.2:44279), f52337e2cb7f4a8d9151f6ee80fabb8e (127.0.98.1:40165)
I20260619 16:45:27.496405 18960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:27.492890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57333 (local address 127.0.98.2:44279)
0619 16:45:27.493078 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:27.493084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:27.493103 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:45:27.493179 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:27.493183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:27.493239 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:27.493348 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:27.493662 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:27.494170 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:27.495313 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:27.495781 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:27.495900 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:27.495968 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:27.496181 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 16:45:27.496261 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 16:45:27.496407 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:27.492717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44279 (local address 127.0.98.3:57333)
0619 16:45:27.492874 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:27.492895 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:27.493041 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:27.493365 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:27.493369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:27.493397 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:27.493533 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:27.493539 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:27.494288 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:27.494293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:27.495165 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:27.495174 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:27.495919 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:27.495923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:27.496046 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:27.496065 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 16:45:27.496150 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 16:45:27.496247 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 16:45:27.496986 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:27.493148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39401 (local address 127.0.98.1:40165)
0619 16:45:27.493295 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:27.493300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:27.493433 (+ 133us) server_negotiation.cc:407] Connection header received
0619 16:45:27.493614 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:27.493617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:27.493666 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:27.493758 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:27.494085 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:27.494637 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:27.495784 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:27.496257 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:27.496393 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:27.496475 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:27.496703 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 16:45:27.496836 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:45:27.497390 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:27.492959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40165 (local address 127.0.98.3:39401)
0619 16:45:27.493311 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:27.493327 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:27.493474 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:27.493769 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:27.493772 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:27.493787 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:27.493956 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:27.493963 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:27.494761 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:27.494768 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:27.495615 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:27.495624 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:27.496396 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:27.496400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:27.496513 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:27.496545 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 16:45:27.497174 (+ 629us) client_negotiation.cc:240] Negotiation successful
0619 16:45:27.497270 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":105,"threads_started":1}
I20260619 16:45:27.497439 18767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee18cda2d98407c81c563bd5239dbac" candidate_uuid: "2aec0408e10740c69d2cd490ee3bbc68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b207254652442118fe0a63878865cde" is_pre_election: true
I20260619 16:45:27.497852 18767 raft_consensus.cc:2454] T cee18cda2d98407c81c563bd5239dbac P 2b207254652442118fe0a63878865cde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aec0408e10740c69d2cd490ee3bbc68 in term 0.
I20260619 16:45:27.498389 18633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee18cda2d98407c81c563bd5239dbac" candidate_uuid: "2aec0408e10740c69d2cd490ee3bbc68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f52337e2cb7f4a8d9151f6ee80fabb8e" is_pre_election: true
I20260619 16:45:27.498418 18837 leader_election.cc:302] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aec0408e10740c69d2cd490ee3bbc68, 2b207254652442118fe0a63878865cde; no voters:
I20260619 16:45:27.498787 18633 raft_consensus.cc:2454] T cee18cda2d98407c81c563bd5239dbac P f52337e2cb7f4a8d9151f6ee80fabb8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aec0408e10740c69d2cd490ee3bbc68 in term 0.
I20260619 16:45:27.498780 18972 raft_consensus.cc:2781] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:45:27.498903 18972 raft_consensus.cc:482] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:45:27.498966 18972 raft_consensus.cc:3037] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:27.500459 18972 raft_consensus.cc:504] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b207254652442118fe0a63878865cde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } } peers { permanent_uuid: "f52337e2cb7f4a8d9151f6ee80fabb8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40165 } } peers { permanent_uuid: "2aec0408e10740c69d2cd490ee3bbc68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34703 } }
I20260619 16:45:27.500919 18972 leader_election.cc:288] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68 [CANDIDATE]: Term 1 election: Requested vote from peers 2b207254652442118fe0a63878865cde (127.0.98.2:44279), f52337e2cb7f4a8d9151f6ee80fabb8e (127.0.98.1:40165)
I20260619 16:45:27.501425 18633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee18cda2d98407c81c563bd5239dbac" candidate_uuid: "2aec0408e10740c69d2cd490ee3bbc68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f52337e2cb7f4a8d9151f6ee80fabb8e"
I20260619 16:45:27.501585 18633 raft_consensus.cc:3037] T cee18cda2d98407c81c563bd5239dbac P f52337e2cb7f4a8d9151f6ee80fabb8e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:27.501731 18767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee18cda2d98407c81c563bd5239dbac" candidate_uuid: "2aec0408e10740c69d2cd490ee3bbc68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b207254652442118fe0a63878865cde"
I20260619 16:45:27.501922 18767 raft_consensus.cc:3037] T cee18cda2d98407c81c563bd5239dbac P 2b207254652442118fe0a63878865cde [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:27.502768 18633 raft_consensus.cc:2454] T cee18cda2d98407c81c563bd5239dbac P f52337e2cb7f4a8d9151f6ee80fabb8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aec0408e10740c69d2cd490ee3bbc68 in term 1.
I20260619 16:45:27.503165 18836 leader_election.cc:302] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aec0408e10740c69d2cd490ee3bbc68, f52337e2cb7f4a8d9151f6ee80fabb8e; no voters:
I20260619 16:45:27.503276 18767 raft_consensus.cc:2454] T cee18cda2d98407c81c563bd5239dbac P 2b207254652442118fe0a63878865cde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aec0408e10740c69d2cd490ee3bbc68 in term 1.
I20260619 16:45:27.503388 18972 raft_consensus.cc:2781] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:27.503749 18972 raft_consensus.cc:686] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68 [term 1 LEADER]: Becoming Leader. State: Replica: 2aec0408e10740c69d2cd490ee3bbc68, State: Running, Role: LEADER
I20260619 16:45:27.504289 18972 consensus_queue.cc:237] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b207254652442118fe0a63878865cde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } } peers { permanent_uuid: "f52337e2cb7f4a8d9151f6ee80fabb8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40165 } } peers { permanent_uuid: "2aec0408e10740c69d2cd490ee3bbc68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34703 } }
I20260619 16:45:27.508543 18487 catalog_manager.cc:5495] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2aec0408e10740c69d2cd490ee3bbc68 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2aec0408e10740c69d2cd490ee3bbc68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b207254652442118fe0a63878865cde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f52337e2cb7f4a8d9151f6ee80fabb8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aec0408e10740c69d2cd490ee3bbc68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34703 } health_report { overall_health: HEALTHY } } }
I20260619 16:45:27.548219 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:27.543921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40262 (local address 127.0.98.3:34703)
0619 16:45:27.544128 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:27.544135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:27.544154 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:45:27.544382 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:27.544385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:27.544457 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:27.544579 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:45:27.544894 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:27.545404 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:27.546255 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:27.546514 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:27.546633 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:27.547156 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:45:27.547617 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:45:27.547835 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:27.547939 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 16:45:27.547984 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260619 16:45:27.556208 18633 raft_consensus.cc:1261] T cee18cda2d98407c81c563bd5239dbac P f52337e2cb7f4a8d9151f6ee80fabb8e [term 1 FOLLOWER]: Refusing update from remote peer 2aec0408e10740c69d2cd490ee3bbc68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:45:27.556208 18767 raft_consensus.cc:1261] T cee18cda2d98407c81c563bd5239dbac P 2b207254652442118fe0a63878865cde [term 1 FOLLOWER]: Refusing update from remote peer 2aec0408e10740c69d2cd490ee3bbc68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:45:27.557344 18974 consensus_queue.cc:1035] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f52337e2cb7f4a8d9151f6ee80fabb8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:45:27.557669 18972 consensus_queue.cc:1035] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b207254652442118fe0a63878865cde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:45:27.567520 18977 mvcc.cc:205] Tried to move back new op lower bound from 7298611312859115520 to 7298611312661078016. Current Snapshot: MvccSnapshot[applied={T|T < 7298611312859115520}]
I20260619 16:45:27.571225 18978 mvcc.cc:205] Tried to move back new op lower bound from 7298611312859115520 to 7298611312661078016. Current Snapshot: MvccSnapshot[applied={T|T < 7298611312859115520}]
I20260619 16:45:27.574823 18979 mvcc.cc:205] Tried to move back new op lower bound from 7298611312859115520 to 7298611312661078016. Current Snapshot: MvccSnapshot[applied={T|T < 7298611312859115520 or (T in {7298611312859115520})}]
I20260619 16:45:27.580188 18486 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:45:27.580539 18486 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:45:27.584007 18486 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4df3efe8e8764e25a747bbe2cf4974f5: Sending DeleteTablet for 3 replicas of tablet cee18cda2d98407c81c563bd5239dbac
I20260619 16:45:27.585255 18613 tablet_service.cc:1514] Processing DeleteTablet for tablet cee18cda2d98407c81c563bd5239dbac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:45:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43320
I20260619 16:45:27.585317 18881 tablet_service.cc:1514] Processing DeleteTablet for tablet cee18cda2d98407c81c563bd5239dbac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:45:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40254
I20260619 16:45:27.586061 18987 tablet_replica.cc:330] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68: stopping tablet replica
I20260619 16:45:27.586256 18747 tablet_service.cc:1514] Processing DeleteTablet for tablet cee18cda2d98407c81c563bd5239dbac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:45:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47118
I20260619 16:45:27.586702 18987 raft_consensus.cc:2229] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:45:27.586793 18989 tablet_replica.cc:330] T cee18cda2d98407c81c563bd5239dbac P 2b207254652442118fe0a63878865cde: stopping tablet replica
I20260619 16:45:27.586910 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 18550
I20260619 16:45:27.586889 18988 tablet_replica.cc:330] T cee18cda2d98407c81c563bd5239dbac P f52337e2cb7f4a8d9151f6ee80fabb8e: stopping tablet replica
I20260619 16:45:27.587102 18989 raft_consensus.cc:2229] T cee18cda2d98407c81c563bd5239dbac P 2b207254652442118fe0a63878865cde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:45:27.587332 18988 raft_consensus.cc:2229] T cee18cda2d98407c81c563bd5239dbac P f52337e2cb7f4a8d9151f6ee80fabb8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:45:27.587810 18989 raft_consensus.cc:2258] T cee18cda2d98407c81c563bd5239dbac P 2b207254652442118fe0a63878865cde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:45:27.589073 18987 raft_consensus.cc:2258] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:45:27.590152 18989 ts_tablet_manager.cc:1905] T cee18cda2d98407c81c563bd5239dbac P 2b207254652442118fe0a63878865cde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:45:27.590561 18987 ts_tablet_manager.cc:1905] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:45:27.593640 18987 ts_tablet_manager.cc:1918] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:45:27.593644 18989 ts_tablet_manager.cc:1918] T cee18cda2d98407c81c563bd5239dbac P 2b207254652442118fe0a63878865cde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:45:27.593762 18989 log.cc:1196] T cee18cda2d98407c81c563bd5239dbac P 2b207254652442118fe0a63878865cde: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-1/wal/wals/cee18cda2d98407c81c563bd5239dbac
I20260619 16:45:27.593925 18987 log.cc:1196] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887259574980-392-0/minicluster-data/ts-2/wal/wals/cee18cda2d98407c81c563bd5239dbac
I20260619 16:45:27.594116 18989 ts_tablet_manager.cc:1939] T cee18cda2d98407c81c563bd5239dbac P 2b207254652442118fe0a63878865cde: Deleting consensus metadata
I20260619 16:45:27.594412 18987 ts_tablet_manager.cc:1939] T cee18cda2d98407c81c563bd5239dbac P 2aec0408e10740c69d2cd490ee3bbc68: Deleting consensus metadata
I20260619 16:45:27.595853 18475 catalog_manager.cc:4841] TS 2b207254652442118fe0a63878865cde (127.0.98.2:44279): tablet cee18cda2d98407c81c563bd5239dbac (table test-table [id=5ec38d387a79400c9cfc68033f8c7098]) successfully deleted
I20260619 16:45:27.595865 18472 catalog_manager.cc:4841] TS 2aec0408e10740c69d2cd490ee3bbc68 (127.0.98.3:34703): tablet cee18cda2d98407c81c563bd5239dbac (table test-table [id=5ec38d387a79400c9cfc68033f8c7098]) successfully deleted
I20260619 16:45:27.596278 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 18684
W20260619 16:45:27.596282 18474 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40165 (error 108)
I20260619 16:45:27.598608 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:27.598123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40165 (local address 127.0.0.1:43334)
0619 16:45:27.598403 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:27.598502 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":103,"threads_started":1}
W20260619 16:45:27.598887 18474 catalog_manager.cc:4584] TS f52337e2cb7f4a8d9151f6ee80fabb8e (127.0.98.1:40165): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cee18cda2d98407c81c563bd5239dbac: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40165: connect: Connection refused (error 111)
I20260619 16:45:27.604321 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 18818
I20260619 16:45:27.613785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 18456
2026-06-19T16:45:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2757 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T16:45:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:45:27Z Disabled control of system clock
I20260619 16:45:27.649448 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43159
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:45:27.787374 18995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:27.787847 18995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:27.787927 18995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:27.792507 18995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:45:27.792655 18995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:27.792683 18995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:27.792717 18995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:45:27.792753 18995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:45:27.798372 18995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43159
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:27.800047 18995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:27.801956 18995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:27.813295 19000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:27.813336 19001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:27.813752 18995 server_base.cc:1034] running on GCE node
W20260619 16:45:27.813774 19003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:27.814649 18995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:27.816078 18995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:27.819764 18995 hybrid_clock.cc:648] HybridClock initialized: now 1781887527819595 us; error 79 us; skew 500 ppm
I20260619 16:45:27.822254 18995 webserver.cc:460] Webserver started at http://127.0.98.62:44537/ using document root <none> and password file <none>
I20260619 16:45:27.822970 18995 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:27.823078 18995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:27.823340 18995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:27.825829 18995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "635f11a745ad469d8ee2469a3daa37f8"
format_stamp: "Formatted at 2026-06-19 16:45:27 on dist-test-slave-03cd"
server_key: "d8a969a820e739e0763380fa1e3ead75"
server_key_iv: "80776809532af7bd3fa2a8f1268d8eef"
server_key_version: "encryptionkey@0"
I20260619 16:45:27.826478 18995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "635f11a745ad469d8ee2469a3daa37f8"
format_stamp: "Formatted at 2026-06-19 16:45:27 on dist-test-slave-03cd"
server_key: "d8a969a820e739e0763380fa1e3ead75"
server_key_iv: "80776809532af7bd3fa2a8f1268d8eef"
server_key_version: "encryptionkey@0"
I20260619 16:45:27.830988 18995 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:45:27.834180 19009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:27.836036 18995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:45:27.836254 18995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "635f11a745ad469d8ee2469a3daa37f8"
format_stamp: "Formatted at 2026-06-19 16:45:27 on dist-test-slave-03cd"
server_key: "d8a969a820e739e0763380fa1e3ead75"
server_key_iv: "80776809532af7bd3fa2a8f1268d8eef"
server_key_version: "encryptionkey@0"
I20260619 16:45:27.836407 18995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:27.851130 18995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:27.852336 18995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:27.852581 18995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:27.860962 18995 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43159
I20260619 16:45:27.861148 19061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43159 every 8 connection(s)
I20260619 16:45:27.862111 18995 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:45:27.866580 19062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:27.868778 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 18995
I20260619 16:45:27.868953 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:45:27.869367 392 external_mini_cluster.cc:1428] Setting key f28343820acd13ca5c19aad03414875f
I20260619 16:45:27.876222 19062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 635f11a745ad469d8ee2469a3daa37f8: Bootstrap starting.
I20260619 16:45:27.879755 19062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 635f11a745ad469d8ee2469a3daa37f8: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:27.880929 19062 log.cc:824] T 00000000000000000000000000000000 P 635f11a745ad469d8ee2469a3daa37f8: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:27.881714 19065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:27.871426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57936 (local address 127.0.98.62:43159)
0619 16:45:27.872484 (+ 1058us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:27.872510 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:27.872572 (+ 62us) server_negotiation.cc:407] Connection header received
0619 16:45:27.873945 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:27.873972 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:27.874500 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:27.874938 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:27.875451 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:27.876615 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:27.877593 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:27.878139 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:27.878335 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:27.879304 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:27.879309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:27.879331 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:27.879912 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:27.880002 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:27.880026 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:27.880139 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:27.880290 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 16:45:27.880618 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":234,"threads_started":1}
I20260619 16:45:27.883461 19062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 635f11a745ad469d8ee2469a3daa37f8: No bootstrap required, opened a new log
I20260619 16:45:27.887840 19062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 635f11a745ad469d8ee2469a3daa37f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635f11a745ad469d8ee2469a3daa37f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43159 } }
I20260619 16:45:27.888186 19062 raft_consensus.cc:374] T 00000000000000000000000000000000 P 635f11a745ad469d8ee2469a3daa37f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:27.888228 19062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 635f11a745ad469d8ee2469a3daa37f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 635f11a745ad469d8ee2469a3daa37f8, State: Initialized, Role: FOLLOWER
I20260619 16:45:27.888725 19062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 635f11a745ad469d8ee2469a3daa37f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635f11a745ad469d8ee2469a3daa37f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43159 } }
I20260619 16:45:27.888998 19062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 635f11a745ad469d8ee2469a3daa37f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:45:27.889143 19062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 635f11a745ad469d8ee2469a3daa37f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:45:27.889276 19062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 635f11a745ad469d8ee2469a3daa37f8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:27.890486 19062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 635f11a745ad469d8ee2469a3daa37f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635f11a745ad469d8ee2469a3daa37f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43159 } }
I20260619 16:45:27.890972 19062 leader_election.cc:302] T 00000000000000000000000000000000 P 635f11a745ad469d8ee2469a3daa37f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 635f11a745ad469d8ee2469a3daa37f8; no voters:
I20260619 16:45:27.891366 19062 leader_election.cc:288] T 00000000000000000000000000000000 P 635f11a745ad469d8ee2469a3daa37f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:45:27.891793 19067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 635f11a745ad469d8ee2469a3daa37f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:27.892179 19067 raft_consensus.cc:686] T 00000000000000000000000000000000 P 635f11a745ad469d8ee2469a3daa37f8 [term 1 LEADER]: Becoming Leader. State: Replica: 635f11a745ad469d8ee2469a3daa37f8, State: Running, Role: LEADER
I20260619 16:45:27.892618 19067 consensus_queue.cc:237] T 00000000000000000000000000000000 P 635f11a745ad469d8ee2469a3daa37f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635f11a745ad469d8ee2469a3daa37f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43159 } }
I20260619 16:45:27.893023 19062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 635f11a745ad469d8ee2469a3daa37f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:45:27.894881 19068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635f11a745ad469d8ee2469a3daa37f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "635f11a745ad469d8ee2469a3daa37f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635f11a745ad469d8ee2469a3daa37f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43159 } } }
I20260619 16:45:27.895107 19068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635f11a745ad469d8ee2469a3daa37f8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:27.895634 19081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:45:27.895615 19069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635f11a745ad469d8ee2469a3daa37f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 635f11a745ad469d8ee2469a3daa37f8. Latest consensus state: current_term: 1 leader_uuid: "635f11a745ad469d8ee2469a3daa37f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635f11a745ad469d8ee2469a3daa37f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43159 } } }
I20260619 16:45:27.896260 19069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635f11a745ad469d8ee2469a3daa37f8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:27.899976 19081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:45:27.907877 19081 catalog_manager.cc:1347] Generated new cluster ID: b1cf99ff674e451389d905a8837aa73e
I20260619 16:45:27.908006 19081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:45:27.915988 19081 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:45:27.917296 19081 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:45:27.928568 19081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 635f11a745ad469d8ee2469a3daa37f8: Generated new TSK 0
I20260619 16:45:27.929626 19081 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:45:27.935711 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43159
--builtin_ntp_servers=127.0.98.20:34059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:45:28.093261 19086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:28.093636 19086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:28.093680 19086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:28.098129 19086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:28.098233 19086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:28.098296 19086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:45:28.103560 19086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:28.104988 19086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:28.106541 19086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:28.117573 19094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:28.117599 19091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:28.117599 19092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:28.117949 19086 server_base.cc:1034] running on GCE node
I20260619 16:45:28.118726 19086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:28.119455 19086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:28.120685 19086 hybrid_clock.cc:648] HybridClock initialized: now 1781887528120652 us; error 59 us; skew 500 ppm
I20260619 16:45:28.122884 19086 webserver.cc:460] Webserver started at http://127.0.98.1:33163/ using document root <none> and password file <none>
I20260619 16:45:28.123505 19086 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:28.123602 19086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:28.123883 19086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:28.126124 19086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-0/data/instance:
uuid: "4ceba2f4f1f24d82ad30f373e1237ec6"
format_stamp: "Formatted at 2026-06-19 16:45:28 on dist-test-slave-03cd"
server_key: "cddf059f374c7e5bf0d63fe5a9622e56"
server_key_iv: "b1caf6fd7832a3f0cae6a89a6da1a163"
server_key_version: "encryptionkey@0"
I20260619 16:45:28.126729 19086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4ceba2f4f1f24d82ad30f373e1237ec6"
format_stamp: "Formatted at 2026-06-19 16:45:28 on dist-test-slave-03cd"
server_key: "cddf059f374c7e5bf0d63fe5a9622e56"
server_key_iv: "b1caf6fd7832a3f0cae6a89a6da1a163"
server_key_version: "encryptionkey@0"
I20260619 16:45:28.131069 19086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 16:45:28.134136 19100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:28.136001 19086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:45:28.136178 19086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "4ceba2f4f1f24d82ad30f373e1237ec6"
format_stamp: "Formatted at 2026-06-19 16:45:28 on dist-test-slave-03cd"
server_key: "cddf059f374c7e5bf0d63fe5a9622e56"
server_key_iv: "b1caf6fd7832a3f0cae6a89a6da1a163"
server_key_version: "encryptionkey@0"
I20260619 16:45:28.136322 19086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:28.160804 19086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:28.161558 19086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:28.161778 19086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:28.162411 19086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:45:28.163483 19086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:45:28.163558 19086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:28.163628 19086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:45:28.163707 19086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260619 16:45:28.174669 19086 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34325
I20260619 16:45:28.174686 19213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34325 every 8 connection(s)
I20260619 16:45:28.175779 19086 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:45:28.185781 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 19086
I20260619 16:45:28.185972 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:45:28.186360 392 external_mini_cluster.cc:1428] Setting key e7f52fb51d665471dafc15cf8348047c
I20260619 16:45:28.188591 19065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:28.178553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41071 (local address 127.0.98.62:43159)
0619 16:45:28.178751 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:28.178761 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:28.180184 (+ 1423us) server_negotiation.cc:407] Connection header received
0619 16:45:28.182073 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:28.182077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:28.182160 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:28.182256 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:28.183482 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.184244 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.185690 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.186018 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.186147 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:28.187945 (+ 1798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:28.187952 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:28.187955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:28.187975 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:28.188000 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:28.188010 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:28.188109 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:28.188309 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 16:45:28.188405 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 16:45:28.188849 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43159
--builtin_ntp_servers=127.0.98.20:34059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:45:28.189859 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:28.178929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43159 (local address 127.0.98.1:41071)
0619 16:45:28.179957 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:28.180025 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:28.181766 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:28.182492 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:28.182501 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:28.182618 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:45:28.183295 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:28.183310 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.184419 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.184422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:28.185532 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:28.185543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.186282 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.186290 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:28.186617 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:28.187519 (+ 902us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:45:28.187521 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:45:28.187592 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:45:28.187762 (+ 170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:45:28.188128 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:45:28.188139 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:45:28.188148 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 16:45:28.188303 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 16:45:28.188725 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":134,"threads_started":1}
I20260619 16:45:28.191799 19214 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43159
I20260619 16:45:28.192139 19214 heartbeater.cc:464] Registering TS with master...
I20260619 16:45:28.193029 19214 heartbeater.cc:511] Master 127.0.98.62:43159 requested a full tablet report, sending...
I20260619 16:45:28.195372 19025 ts_manager.cc:194] Registered new tserver with Master: 4ceba2f4f1f24d82ad30f373e1237ec6 (127.0.98.1:34325)
I20260619 16:45:28.196862 19025 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:41071
W20260619 16:45:28.305509 19217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:28.305816 19217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:28.305857 19217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:28.310204 19217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:28.310288 19217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:28.310348 19217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:45:28.315605 19217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:28.316902 19217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:28.318398 19217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:28.329772 19223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:28.329854 19217 server_base.cc:1034] running on GCE node
W20260619 16:45:28.329803 19225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:28.329803 19222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:28.330631 19217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:28.331533 19217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:28.332772 19217 hybrid_clock.cc:648] HybridClock initialized: now 1781887528332738 us; error 57 us; skew 500 ppm
I20260619 16:45:28.335505 19217 webserver.cc:460] Webserver started at http://127.0.98.2:35435/ using document root <none> and password file <none>
I20260619 16:45:28.336380 19217 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:28.336501 19217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:28.336805 19217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:28.339761 19217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-1/data/instance:
uuid: "f5e92bfd1ae34488a3eb6823c1d81b7f"
format_stamp: "Formatted at 2026-06-19 16:45:28 on dist-test-slave-03cd"
server_key: "1098840a766d9f899b5e26eb1750e2f7"
server_key_iv: "dfcf046dbe7b2480f1b65b9d9170be49"
server_key_version: "encryptionkey@0"
I20260619 16:45:28.340588 19217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f5e92bfd1ae34488a3eb6823c1d81b7f"
format_stamp: "Formatted at 2026-06-19 16:45:28 on dist-test-slave-03cd"
server_key: "1098840a766d9f899b5e26eb1750e2f7"
server_key_iv: "dfcf046dbe7b2480f1b65b9d9170be49"
server_key_version: "encryptionkey@0"
I20260619 16:45:28.345698 19217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:45:28.349578 19231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:28.351634 19217 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:45:28.351975 19217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "f5e92bfd1ae34488a3eb6823c1d81b7f"
format_stamp: "Formatted at 2026-06-19 16:45:28 on dist-test-slave-03cd"
server_key: "1098840a766d9f899b5e26eb1750e2f7"
server_key_iv: "dfcf046dbe7b2480f1b65b9d9170be49"
server_key_version: "encryptionkey@0"
I20260619 16:45:28.352203 19217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:28.376209 19217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:28.377166 19217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:28.377429 19217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:28.378211 19217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:45:28.379442 19217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:45:28.379498 19217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:28.379585 19217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:45:28.379621 19217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:28.391821 19217 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39211
I20260619 16:45:28.391981 19344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39211 every 8 connection(s)
I20260619 16:45:28.393137 19217 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:45:28.396924 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 19217
I20260619 16:45:28.397059 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:45:28.397439 392 external_mini_cluster.cc:1428] Setting key 3ab2ae205c47b5a3b1740cc13d7ac8dd
I20260619 16:45:28.400260 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43159
--builtin_ntp_servers=127.0.98.20:34059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:45:28.405476 19065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:28.395483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51579 (local address 127.0.98.62:43159)
0619 16:45:28.395630 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:28.395635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:28.397096 (+ 1461us) server_negotiation.cc:407] Connection header received
0619 16:45:28.399131 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:28.399137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:28.399233 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:28.399349 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:28.400927 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.401469 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.402968 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.403214 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.403335 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:28.404888 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:28.404891 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:28.404894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:28.404916 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:28.404936 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:28.404945 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:28.405039 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:28.405220 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 16:45:28.405310 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:45:28.406474 19347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:28.396021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43159 (local address 127.0.98.2:51579)
0619 16:45:28.396902 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:28.396964 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:28.398739 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:28.399688 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:28.399701 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:28.399843 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:45:28.400739 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:28.400767 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.401617 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.401625 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:28.402804 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:28.402829 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.403419 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.403426 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:28.403707 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:28.404509 (+ 802us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:45:28.404511 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:45:28.404557 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:45:28.404632 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:45:28.405043 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:45:28.405054 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:45:28.405061 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 16:45:28.405203 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 16:45:28.405524 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":137,"threads_started":1}
I20260619 16:45:28.407940 19345 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43159
I20260619 16:45:28.408215 19345 heartbeater.cc:464] Registering TS with master...
I20260619 16:45:28.408809 19345 heartbeater.cc:511] Master 127.0.98.62:43159 requested a full tablet report, sending...
I20260619 16:45:28.409973 19025 ts_manager.cc:194] Registered new tserver with Master: f5e92bfd1ae34488a3eb6823c1d81b7f (127.0.98.2:39211)
I20260619 16:45:28.410593 19025 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:51579
W20260619 16:45:28.519201 19348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:28.519527 19348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:28.519605 19348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:28.523923 19348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:28.524017 19348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:28.524106 19348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:45:28.529511 19348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:28.530697 19348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:28.532220 19348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:28.542842 19354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:28.542960 19348 server_base.cc:1034] running on GCE node
W20260619 16:45:28.542815 19356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:28.542846 19353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:28.543720 19348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:28.544441 19348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:28.545647 19348 hybrid_clock.cc:648] HybridClock initialized: now 1781887528545622 us; error 41 us; skew 500 ppm
I20260619 16:45:28.547952 19348 webserver.cc:460] Webserver started at http://127.0.98.3:44155/ using document root <none> and password file <none>
I20260619 16:45:28.548657 19348 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:28.548749 19348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:28.548972 19348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:28.551151 19348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-2/data/instance:
uuid: "62894719b16f4cf18a2485e3804f5c0e"
format_stamp: "Formatted at 2026-06-19 16:45:28 on dist-test-slave-03cd"
server_key: "472a5509dd5558854f0699b7212169c8"
server_key_iv: "564c77b12f36d440002ac9c77d77571d"
server_key_version: "encryptionkey@0"
I20260619 16:45:28.551812 19348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-2/wal/instance:
uuid: "62894719b16f4cf18a2485e3804f5c0e"
format_stamp: "Formatted at 2026-06-19 16:45:28 on dist-test-slave-03cd"
server_key: "472a5509dd5558854f0699b7212169c8"
server_key_iv: "564c77b12f36d440002ac9c77d77571d"
server_key_version: "encryptionkey@0"
I20260619 16:45:28.555879 19348 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:45:28.558822 19362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:28.560307 19348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:45:28.560467 19348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "62894719b16f4cf18a2485e3804f5c0e"
format_stamp: "Formatted at 2026-06-19 16:45:28 on dist-test-slave-03cd"
server_key: "472a5509dd5558854f0699b7212169c8"
server_key_iv: "564c77b12f36d440002ac9c77d77571d"
server_key_version: "encryptionkey@0"
I20260619 16:45:28.560598 19348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:28.583916 19348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:28.584705 19348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:28.584918 19348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:28.585546 19348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:45:28.586611 19348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:45:28.586684 19348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:28.586755 19348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:45:28.586802 19348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:28.597256 19348 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46173
I20260619 16:45:28.597280 19475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46173 every 8 connection(s)
I20260619 16:45:28.598315 19348 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:45:28.608469 19065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:28.600431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59977 (local address 127.0.98.62:43159)
0619 16:45:28.600592 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:28.600596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:28.601582 (+ 986us) server_negotiation.cc:407] Connection header received
0619 16:45:28.603179 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:28.603183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:28.603248 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:28.603364 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:28.604375 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.604919 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.606093 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.606328 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.606448 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:28.607735 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:28.607805 (+ 70us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:28.607809 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:28.607831 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:28.607856 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:28.607869 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:28.608000 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:28.608212 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 16:45:28.608293 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:45:28.608498 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 19348
I20260619 16:45:28.608645 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:45:28.608997 392 external_mini_cluster.cc:1428] Setting key 6d007f23f77f72af652cb39d0b0b43e2
I20260619 16:45:28.609532 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:28.600736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43159 (local address 127.0.98.3:59977)
0619 16:45:28.601413 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:28.601456 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:28.602948 (+ 1492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:28.603538 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:28.603545 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:28.603633 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:45:28.604231 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:28.604244 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.605061 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.605064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:28.605964 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:28.605974 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.606495 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.606499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:28.606670 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:28.607352 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:45:28.607354 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:45:28.607402 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:45:28.607480 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:45:28.608002 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:45:28.608015 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:45:28.608025 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 16:45:28.608212 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 16:45:28.608559 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":111,"threads_started":1}
I20260619 16:45:28.610762 19476 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43159
I20260619 16:45:28.611068 19476 heartbeater.cc:464] Registering TS with master...
I20260619 16:45:28.611722 19476 heartbeater.cc:511] Master 127.0.98.62:43159 requested a full tablet report, sending...
I20260619 16:45:28.612869 19025 ts_manager.cc:194] Registered new tserver with Master: 62894719b16f4cf18a2485e3804f5c0e (127.0.98.3:46173)
I20260619 16:45:28.613502 19025 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:59977
I20260619 16:45:28.623641 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:45:28.630604 19065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:28.625973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57950 (local address 127.0.98.62:43159)
0619 16:45:28.626135 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:28.626139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:28.626200 (+ 61us) server_negotiation.cc:407] Connection header received
0619 16:45:28.626386 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:28.626389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:28.626436 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:28.626513 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:28.626898 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.627456 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.628593 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.628823 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.628915 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:28.629950 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:28.629953 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:28.629956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:28.629970 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:28.629991 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:28.629999 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:28.630102 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:28.630351 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 16:45:28.630404 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 16:45:28.633841 19025 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:45:28.637218 19025 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:45:28.653230 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:28.647487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40430 (local address 127.0.98.2:39211)
0619 16:45:28.647894 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:28.647906 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:28.648287 (+ 381us) server_negotiation.cc:407] Connection header received
0619 16:45:28.648383 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:28.648388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:28.648584 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:28.649130 (+ 546us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:28.649232 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.650382 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.651975 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.652554 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.652685 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:28.652906 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:28.652962 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 16:45:28.653057 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":118,"threads_started":1}
I20260619 16:45:28.653229 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:28.646916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39211 (local address 127.0.0.1:40430)
0619 16:45:28.647249 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:28.647301 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:28.647453 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:28.648737 (+ 1284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:28.648740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:28.648772 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:28.648978 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:28.648986 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.650508 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.650511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:28.651840 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:28.651848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.652686 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.652690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:28.652792 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:28.652827 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 16:45:28.652933 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 16:45:28.653052 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":16,"spinlock_wait_cycles":12032,"thread_start_us":128,"threads_started":1}
I20260619 16:45:28.653533 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:28.646742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46173 (local address 127.0.0.1:60346)
0619 16:45:28.647249 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:28.647302 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:28.647453 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:28.648696 (+ 1243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:28.648703 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:28.648739 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:28.648978 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:28.648986 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.650162 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.650169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:28.651837 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:28.651848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.653080 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.653084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:28.653180 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:28.653204 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 16:45:28.653264 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 16:45:28.653351 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":173,"spinlock_wait_cycles":3072,"thread_start_us":107,"threads_started":1}
I20260619 16:45:28.654174 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:28.647467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34325 (local address 127.0.0.1:47806)
0619 16:45:28.648349 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:28.648363 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:28.648518 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:28.649173 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:28.649176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:28.649194 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:28.649322 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:28.649329 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.650652 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.650656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:28.651860 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:28.651870 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.653743 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.653748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:28.653853 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:28.653878 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 16:45:28.653967 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 16:45:28.654039 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":45,"thread_start_us":86,"threads_started":1}
I20260619 16:45:28.654422 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:28.647947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47806 (local address 127.0.98.1:34325)
0619 16:45:28.648219 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:28.648226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:28.648451 (+ 225us) server_negotiation.cc:407] Connection header received
0619 16:45:28.648687 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:28.648692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:28.648875 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:28.648993 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:28.649451 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.650534 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.652050 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.652912 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.653049 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:28.653246 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:28.654002 (+ 756us) server_negotiation.cc:299] Negotiation successful
0619 16:45:28.654146 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":70,"threads_started":1}
I20260619 16:45:28.656348 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:28.647728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60346 (local address 127.0.98.3:46173)
0619 16:45:28.648148 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:28.648160 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:28.648184 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:45:28.648273 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:28.648280 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:28.648518 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:28.648905 (+ 387us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:28.649079 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.650033 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.652045 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.652912 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.655798 (+ 2886us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:28.656006 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:28.656060 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 16:45:28.656162 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":157,"threads_started":1}
I20260619 16:45:28.656826 19148 tablet_service.cc:1467] Processing CreateTablet for tablet 9f87be3653fc4a678d25aaf5a19caf5a (DEFAULT_TABLE table=test-table [id=58adab370075418b8f8661c0df9a0c62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:45:28.656894 19279 tablet_service.cc:1467] Processing CreateTablet for tablet 9f87be3653fc4a678d25aaf5a19caf5a (DEFAULT_TABLE table=test-table [id=58adab370075418b8f8661c0df9a0c62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:45:28.657938 19279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f87be3653fc4a678d25aaf5a19caf5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:28.657944 19148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f87be3653fc4a678d25aaf5a19caf5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:28.659041 19410 tablet_service.cc:1467] Processing CreateTablet for tablet 9f87be3653fc4a678d25aaf5a19caf5a (DEFAULT_TABLE table=test-table [id=58adab370075418b8f8661c0df9a0c62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:45:28.660107 19410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f87be3653fc4a678d25aaf5a19caf5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:28.664867 19493 tablet_bootstrap.cc:492] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f: Bootstrap starting.
I20260619 16:45:28.665714 19494 tablet_bootstrap.cc:492] T 9f87be3653fc4a678d25aaf5a19caf5a P 4ceba2f4f1f24d82ad30f373e1237ec6: Bootstrap starting.
I20260619 16:45:28.667433 19493 tablet_bootstrap.cc:654] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:28.667493 19495 tablet_bootstrap.cc:492] T 9f87be3653fc4a678d25aaf5a19caf5a P 62894719b16f4cf18a2485e3804f5c0e: Bootstrap starting.
I20260619 16:45:28.667867 19494 tablet_bootstrap.cc:654] T 9f87be3653fc4a678d25aaf5a19caf5a P 4ceba2f4f1f24d82ad30f373e1237ec6: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:28.668609 19494 log.cc:824] T 9f87be3653fc4a678d25aaf5a19caf5a P 4ceba2f4f1f24d82ad30f373e1237ec6: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:28.668658 19493 log.cc:824] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:28.669765 19495 tablet_bootstrap.cc:654] T 9f87be3653fc4a678d25aaf5a19caf5a P 62894719b16f4cf18a2485e3804f5c0e: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:28.670662 19495 log.cc:824] T 9f87be3653fc4a678d25aaf5a19caf5a P 62894719b16f4cf18a2485e3804f5c0e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:28.670814 19494 tablet_bootstrap.cc:492] T 9f87be3653fc4a678d25aaf5a19caf5a P 4ceba2f4f1f24d82ad30f373e1237ec6: No bootstrap required, opened a new log
I20260619 16:45:28.671065 19494 ts_tablet_manager.cc:1397] T 9f87be3653fc4a678d25aaf5a19caf5a P 4ceba2f4f1f24d82ad30f373e1237ec6: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 16:45:28.671343 19493 tablet_bootstrap.cc:492] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f: No bootstrap required, opened a new log
I20260619 16:45:28.671557 19493 ts_tablet_manager.cc:1397] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 16:45:28.672717 19495 tablet_bootstrap.cc:492] T 9f87be3653fc4a678d25aaf5a19caf5a P 62894719b16f4cf18a2485e3804f5c0e: No bootstrap required, opened a new log
I20260619 16:45:28.672935 19495 ts_tablet_manager.cc:1397] T 9f87be3653fc4a678d25aaf5a19caf5a P 62894719b16f4cf18a2485e3804f5c0e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 16:45:28.675642 19493 raft_consensus.cc:348] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62894719b16f4cf18a2485e3804f5c0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 } } peers { permanent_uuid: "f5e92bfd1ae34488a3eb6823c1d81b7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39211 } } peers { permanent_uuid: "4ceba2f4f1f24d82ad30f373e1237ec6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34325 } }
I20260619 16:45:28.675650 19494 raft_consensus.cc:348] T 9f87be3653fc4a678d25aaf5a19caf5a P 4ceba2f4f1f24d82ad30f373e1237ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62894719b16f4cf18a2485e3804f5c0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 } } peers { permanent_uuid: "f5e92bfd1ae34488a3eb6823c1d81b7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39211 } } peers { permanent_uuid: "4ceba2f4f1f24d82ad30f373e1237ec6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34325 } }
I20260619 16:45:28.675985 19493 raft_consensus.cc:374] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:28.675985 19494 raft_consensus.cc:374] T 9f87be3653fc4a678d25aaf5a19caf5a P 4ceba2f4f1f24d82ad30f373e1237ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:28.676046 19493 raft_consensus.cc:729] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e92bfd1ae34488a3eb6823c1d81b7f, State: Initialized, Role: FOLLOWER
I20260619 16:45:28.676046 19494 raft_consensus.cc:729] T 9f87be3653fc4a678d25aaf5a19caf5a P 4ceba2f4f1f24d82ad30f373e1237ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ceba2f4f1f24d82ad30f373e1237ec6, State: Initialized, Role: FOLLOWER
I20260619 16:45:28.676548 19493 consensus_queue.cc:260] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62894719b16f4cf18a2485e3804f5c0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 } } peers { permanent_uuid: "f5e92bfd1ae34488a3eb6823c1d81b7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39211 } } peers { permanent_uuid: "4ceba2f4f1f24d82ad30f373e1237ec6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34325 } }
I20260619 16:45:28.676548 19494 consensus_queue.cc:260] T 9f87be3653fc4a678d25aaf5a19caf5a P 4ceba2f4f1f24d82ad30f373e1237ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62894719b16f4cf18a2485e3804f5c0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 } } peers { permanent_uuid: "f5e92bfd1ae34488a3eb6823c1d81b7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39211 } } peers { permanent_uuid: "4ceba2f4f1f24d82ad30f373e1237ec6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34325 } }
I20260619 16:45:28.676699 19495 raft_consensus.cc:348] T 9f87be3653fc4a678d25aaf5a19caf5a P 62894719b16f4cf18a2485e3804f5c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62894719b16f4cf18a2485e3804f5c0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 } } peers { permanent_uuid: "f5e92bfd1ae34488a3eb6823c1d81b7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39211 } } peers { permanent_uuid: "4ceba2f4f1f24d82ad30f373e1237ec6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34325 } }
I20260619 16:45:28.676962 19495 raft_consensus.cc:374] T 9f87be3653fc4a678d25aaf5a19caf5a P 62894719b16f4cf18a2485e3804f5c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:28.677018 19495 raft_consensus.cc:729] T 9f87be3653fc4a678d25aaf5a19caf5a P 62894719b16f4cf18a2485e3804f5c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62894719b16f4cf18a2485e3804f5c0e, State: Initialized, Role: FOLLOWER
I20260619 16:45:28.677480 19495 consensus_queue.cc:260] T 9f87be3653fc4a678d25aaf5a19caf5a P 62894719b16f4cf18a2485e3804f5c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62894719b16f4cf18a2485e3804f5c0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 } } peers { permanent_uuid: "f5e92bfd1ae34488a3eb6823c1d81b7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39211 } } peers { permanent_uuid: "4ceba2f4f1f24d82ad30f373e1237ec6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34325 } }
I20260619 16:45:28.677665 19494 ts_tablet_manager.cc:1428] T 9f87be3653fc4a678d25aaf5a19caf5a P 4ceba2f4f1f24d82ad30f373e1237ec6: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260619 16:45:28.677660 19493 ts_tablet_manager.cc:1428] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260619 16:45:28.678026 19345 heartbeater.cc:503] Master 127.0.98.62:43159 was elected leader, sending a full tablet report...
I20260619 16:45:28.678040 19214 heartbeater.cc:503] Master 127.0.98.62:43159 was elected leader, sending a full tablet report...
I20260619 16:45:28.678521 19476 heartbeater.cc:503] Master 127.0.98.62:43159 was elected leader, sending a full tablet report...
I20260619 16:45:28.679087 19495 ts_tablet_manager.cc:1428] T 9f87be3653fc4a678d25aaf5a19caf5a P 62894719b16f4cf18a2485e3804f5c0e: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260619 16:45:28.729067 19499 raft_consensus.cc:482] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:45:28.729286 19499 raft_consensus.cc:504] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62894719b16f4cf18a2485e3804f5c0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 } } peers { permanent_uuid: "f5e92bfd1ae34488a3eb6823c1d81b7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39211 } } peers { permanent_uuid: "4ceba2f4f1f24d82ad30f373e1237ec6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34325 } }
I20260619 16:45:28.730674 19499 leader_election.cc:288] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62894719b16f4cf18a2485e3804f5c0e (127.0.98.3:46173), 4ceba2f4f1f24d82ad30f373e1237ec6 (127.0.98.1:34325)
I20260619 16:45:28.736281 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:28.731280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55425 (local address 127.0.98.3:46173)
0619 16:45:28.731416 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:28.731421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:28.731618 (+ 197us) server_negotiation.cc:407] Connection header received
0619 16:45:28.731735 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:28.731738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:28.731788 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:28.731884 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:28.732144 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.732965 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.734827 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.735528 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.735726 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:28.735789 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:28.736079 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 16:45:28.736160 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:45:28.736326 19347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:28.730998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34325 (local address 127.0.98.2:45525)
0619 16:45:28.731140 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:28.731159 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:28.731264 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:28.731759 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:28.731761 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:28.731774 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:28.731932 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:28.731940 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.733092 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.733095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:28.734645 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:28.734658 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.735714 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.735718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:28.735866 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:28.735952 (+ 86us) client_negotiation.cc:769] Sending connection context
0619 16:45:28.736041 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 16:45:28.736148 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":182}
I20260619 16:45:28.736334 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:28.730998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46173 (local address 127.0.98.2:55425)
0619 16:45:28.731517 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:28.731528 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:28.731610 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:28.731900 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:28.731903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:28.731916 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:28.732025 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:28.732030 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.733092 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.733095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:28.734645 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:28.734658 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.735714 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.735718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:28.735887 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:28.735954 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 16:45:28.736041 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 16:45:28.736148 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":408,"thread_start_us":165,"threads_started":1}
I20260619 16:45:28.736768 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:28.731281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45525 (local address 127.0.98.1:34325)
0619 16:45:28.731416 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:28.731421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:28.731441 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:45:28.731512 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:28.731515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:28.731562 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:28.731741 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:28.732075 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.732917 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.734827 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.735528 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.735686 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:28.735761 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:28.736535 (+ 774us) server_negotiation.cc:299] Negotiation successful
0619 16:45:28.736642 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:45:28.737314 19430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f87be3653fc4a678d25aaf5a19caf5a" candidate_uuid: "f5e92bfd1ae34488a3eb6823c1d81b7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62894719b16f4cf18a2485e3804f5c0e" is_pre_election: true
I20260619 16:45:28.737342 19168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f87be3653fc4a678d25aaf5a19caf5a" candidate_uuid: "f5e92bfd1ae34488a3eb6823c1d81b7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ceba2f4f1f24d82ad30f373e1237ec6" is_pre_election: true
I20260619 16:45:28.737680 19168 raft_consensus.cc:2454] T 9f87be3653fc4a678d25aaf5a19caf5a P 4ceba2f4f1f24d82ad30f373e1237ec6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5e92bfd1ae34488a3eb6823c1d81b7f in term 0.
I20260619 16:45:28.737680 19430 raft_consensus.cc:2454] T 9f87be3653fc4a678d25aaf5a19caf5a P 62894719b16f4cf18a2485e3804f5c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5e92bfd1ae34488a3eb6823c1d81b7f in term 0.
I20260619 16:45:28.738258 19232 leader_election.cc:302] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4ceba2f4f1f24d82ad30f373e1237ec6, 62894719b16f4cf18a2485e3804f5c0e, f5e92bfd1ae34488a3eb6823c1d81b7f; no voters:
I20260619 16:45:28.738574 19499 raft_consensus.cc:2781] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:45:28.738669 19499 raft_consensus.cc:482] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:45:28.738739 19499 raft_consensus.cc:3037] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:28.739867 19499 raft_consensus.cc:504] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62894719b16f4cf18a2485e3804f5c0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 } } peers { permanent_uuid: "f5e92bfd1ae34488a3eb6823c1d81b7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39211 } } peers { permanent_uuid: "4ceba2f4f1f24d82ad30f373e1237ec6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34325 } }
I20260619 16:45:28.740324 19499 leader_election.cc:288] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f [CANDIDATE]: Term 1 election: Requested vote from peers 62894719b16f4cf18a2485e3804f5c0e (127.0.98.3:46173), 4ceba2f4f1f24d82ad30f373e1237ec6 (127.0.98.1:34325)
I20260619 16:45:28.740759 19430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f87be3653fc4a678d25aaf5a19caf5a" candidate_uuid: "f5e92bfd1ae34488a3eb6823c1d81b7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62894719b16f4cf18a2485e3804f5c0e"
I20260619 16:45:28.740759 19168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f87be3653fc4a678d25aaf5a19caf5a" candidate_uuid: "f5e92bfd1ae34488a3eb6823c1d81b7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ceba2f4f1f24d82ad30f373e1237ec6"
I20260619 16:45:28.740909 19430 raft_consensus.cc:3037] T 9f87be3653fc4a678d25aaf5a19caf5a P 62894719b16f4cf18a2485e3804f5c0e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:28.740952 19168 raft_consensus.cc:3037] T 9f87be3653fc4a678d25aaf5a19caf5a P 4ceba2f4f1f24d82ad30f373e1237ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:28.742305 19430 raft_consensus.cc:2454] T 9f87be3653fc4a678d25aaf5a19caf5a P 62894719b16f4cf18a2485e3804f5c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5e92bfd1ae34488a3eb6823c1d81b7f in term 1.
I20260619 16:45:28.742301 19168 raft_consensus.cc:2454] T 9f87be3653fc4a678d25aaf5a19caf5a P 4ceba2f4f1f24d82ad30f373e1237ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5e92bfd1ae34488a3eb6823c1d81b7f in term 1.
I20260619 16:45:28.742643 19232 leader_election.cc:302] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62894719b16f4cf18a2485e3804f5c0e, f5e92bfd1ae34488a3eb6823c1d81b7f; no voters:
I20260619 16:45:28.742834 19499 raft_consensus.cc:2781] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:28.743096 19499 raft_consensus.cc:686] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f [term 1 LEADER]: Becoming Leader. State: Replica: f5e92bfd1ae34488a3eb6823c1d81b7f, State: Running, Role: LEADER
I20260619 16:45:28.743463 19499 consensus_queue.cc:237] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62894719b16f4cf18a2485e3804f5c0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 } } peers { permanent_uuid: "f5e92bfd1ae34488a3eb6823c1d81b7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39211 } } peers { permanent_uuid: "4ceba2f4f1f24d82ad30f373e1237ec6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34325 } }
I20260619 16:45:28.746939 19023 catalog_manager.cc:5495] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f reported cstate change: term changed from 0 to 1, leader changed from <none> to f5e92bfd1ae34488a3eb6823c1d81b7f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f5e92bfd1ae34488a3eb6823c1d81b7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62894719b16f4cf18a2485e3804f5c0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5e92bfd1ae34488a3eb6823c1d81b7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ceba2f4f1f24d82ad30f373e1237ec6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34325 } health_report { overall_health: UNKNOWN } } }
I20260619 16:45:28.779870 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:28.775944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40434 (local address 127.0.98.2:39211)
0619 16:45:28.776081 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:28.776084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:28.776097 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:45:28.776169 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:28.776171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:28.776226 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:28.776330 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:45:28.776624 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.777181 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.778122 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:28.778359 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:28.778464 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:28.778829 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:45:28.779270 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:45:28.779389 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:28.779572 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 16:45:28.779612 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:45:28.786334 19168 raft_consensus.cc:1261] T 9f87be3653fc4a678d25aaf5a19caf5a P 4ceba2f4f1f24d82ad30f373e1237ec6 [term 1 FOLLOWER]: Refusing update from remote peer f5e92bfd1ae34488a3eb6823c1d81b7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:45:28.786334 19430 raft_consensus.cc:1261] T 9f87be3653fc4a678d25aaf5a19caf5a P 62894719b16f4cf18a2485e3804f5c0e [term 1 FOLLOWER]: Refusing update from remote peer f5e92bfd1ae34488a3eb6823c1d81b7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:45:28.787362 19503 consensus_queue.cc:1035] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ceba2f4f1f24d82ad30f373e1237ec6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:45:28.787626 19499 consensus_queue.cc:1035] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "62894719b16f4cf18a2485e3804f5c0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:45:28.795776 19506 mvcc.cc:205] Tried to move back new op lower bound from 7298611317900517376 to 7298611317736366080. Current Snapshot: MvccSnapshot[applied={T|T < 7298611317900517376}]
I20260619 16:45:28.798895 19507 mvcc.cc:205] Tried to move back new op lower bound from 7298611317900517376 to 7298611317736366080. Current Snapshot: MvccSnapshot[applied={T|T < 7298611317900517376}]
I20260619 16:45:28.805732 19025 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:45:28.805953 19025 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:45:28.809103 19025 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 635f11a745ad469d8ee2469a3daa37f8: Sending DeleteTablet for 3 replicas of tablet 9f87be3653fc4a678d25aaf5a19caf5a
I20260619 16:45:28.810174 19410 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f87be3653fc4a678d25aaf5a19caf5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:45:28 UTC) from {username='slave'} at 127.0.0.1:60346
I20260619 16:45:28.810175 19279 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f87be3653fc4a678d25aaf5a19caf5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:45:28 UTC) from {username='slave'} at 127.0.0.1:40430
I20260619 16:45:28.810349 19148 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f87be3653fc4a678d25aaf5a19caf5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:45:28 UTC) from {username='slave'} at 127.0.0.1:47806
I20260619 16:45:28.810763 19516 tablet_replica.cc:330] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f: stopping tablet replica
I20260619 16:45:28.811103 19516 raft_consensus.cc:2229] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:45:28.811548 19517 tablet_replica.cc:330] T 9f87be3653fc4a678d25aaf5a19caf5a P 62894719b16f4cf18a2485e3804f5c0e: stopping tablet replica
I20260619 16:45:28.811584 19516 raft_consensus.cc:2258] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:45:28.811730 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 19086
I20260619 16:45:28.811884 19517 raft_consensus.cc:2229] T 9f87be3653fc4a678d25aaf5a19caf5a P 62894719b16f4cf18a2485e3804f5c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:45:28.812201 19517 raft_consensus.cc:2258] T 9f87be3653fc4a678d25aaf5a19caf5a P 62894719b16f4cf18a2485e3804f5c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:45:28.813833 19517 ts_tablet_manager.cc:1905] T 9f87be3653fc4a678d25aaf5a19caf5a P 62894719b16f4cf18a2485e3804f5c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:45:28.814143 19516 ts_tablet_manager.cc:1905] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:45:28.816828 19517 ts_tablet_manager.cc:1918] T 9f87be3653fc4a678d25aaf5a19caf5a P 62894719b16f4cf18a2485e3804f5c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:45:28.816948 19517 log.cc:1196] T 9f87be3653fc4a678d25aaf5a19caf5a P 62894719b16f4cf18a2485e3804f5c0e: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-2/wal/wals/9f87be3653fc4a678d25aaf5a19caf5a
I20260619 16:45:28.817288 19517 ts_tablet_manager.cc:1939] T 9f87be3653fc4a678d25aaf5a19caf5a P 62894719b16f4cf18a2485e3804f5c0e: Deleting consensus metadata
I20260619 16:45:28.817421 19516 ts_tablet_manager.cc:1918] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:45:28.817526 19516 log.cc:1196] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887259574980-392-0/minicluster-data/ts-1/wal/wals/9f87be3653fc4a678d25aaf5a19caf5a
I20260619 16:45:28.817888 19516 ts_tablet_manager.cc:1939] T 9f87be3653fc4a678d25aaf5a19caf5a P f5e92bfd1ae34488a3eb6823c1d81b7f: Deleting consensus metadata
I20260619 16:45:28.820298 19013 catalog_manager.cc:4841] TS f5e92bfd1ae34488a3eb6823c1d81b7f (127.0.98.2:39211): tablet 9f87be3653fc4a678d25aaf5a19caf5a (table test-table [id=58adab370075418b8f8661c0df9a0c62]) successfully deleted
I20260619 16:45:28.821491 19010 catalog_manager.cc:4841] TS 62894719b16f4cf18a2485e3804f5c0e (127.0.98.3:46173): tablet 9f87be3653fc4a678d25aaf5a19caf5a (table test-table [id=58adab370075418b8f8661c0df9a0c62]) successfully deleted
W20260619 16:45:28.824774 19013 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34325 (error 108)
I20260619 16:45:28.825652 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 19217
I20260619 16:45:28.826926 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:28.825812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34325 (local address 127.0.0.1:47808)
0619 16:45:28.826598 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:28.826799 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":661}
W20260619 16:45:28.827425 19013 catalog_manager.cc:4584] TS 4ceba2f4f1f24d82ad30f373e1237ec6 (127.0.98.1:34325): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9f87be3653fc4a678d25aaf5a19caf5a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34325: connect: Connection refused (error 111)
I20260619 16:45:28.833066 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 19348
I20260619 16:45:28.838742 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 18995
2026-06-19T16:45:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1222 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:45:28 dist-test-slave-03cd krb5kdc[19522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:45:28 dist-test-slave-03cd krb5kdc[19522](info): set up 2 sockets
Jun 19 16:45:28 dist-test-slave-03cd krb5kdc[19522](info): commencing operation
krb5kdc: starting...
W20260619 16:45:29.904129 392 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:45:29 dist-test-slave-03cd krb5kdc[19522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:45:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:45:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:45:30.048369 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46113
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:30.164031 19535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:30.164376 19535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:30.164420 19535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:30.169512 19535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:45:30.169611 19535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:30.169633 19535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:30.169651 19535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:45:30.169667 19535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:45:30.176160 19535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:30.177649 19535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:30.179019 19535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:30.188526 19540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:30.188532 19541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:30.188792 19535 server_base.cc:1034] running on GCE node
W20260619 16:45:30.188532 19543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:30.189770 19535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:30.191093 19535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:30.192337 19535 hybrid_clock.cc:648] HybridClock initialized: now 1781887530192308 us; error 50 us; skew 500 ppm
Jun 19 16:45:30 dist-test-slave-03cd krb5kdc[19522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:30.196154 19535 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:45:30.197469 19535 webserver.cc:460] Webserver started at http://127.0.98.62:34207/ using document root <none> and password file <none>
I20260619 16:45:30.198150 19535 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:30.198232 19535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:30.198491 19535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:30.200912 19535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "ab0c781ac6e54b84b4c9047d8cad0386"
format_stamp: "Formatted at 2026-06-19 16:45:30 on dist-test-slave-03cd"
server_key: "b0e37ca7fb910e90e56d1926816dc680"
server_key_iv: "636e77db4bb43d008d712430e98079b5"
server_key_version: "encryptionkey@0"
I20260619 16:45:30.201572 19535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "ab0c781ac6e54b84b4c9047d8cad0386"
format_stamp: "Formatted at 2026-06-19 16:45:30 on dist-test-slave-03cd"
server_key: "b0e37ca7fb910e90e56d1926816dc680"
server_key_iv: "636e77db4bb43d008d712430e98079b5"
server_key_version: "encryptionkey@0"
I20260619 16:45:30.206602 19535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 16:45:30.209494 19550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:30.211000 19535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:45:30.211155 19535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "ab0c781ac6e54b84b4c9047d8cad0386"
format_stamp: "Formatted at 2026-06-19 16:45:30 on dist-test-slave-03cd"
server_key: "b0e37ca7fb910e90e56d1926816dc680"
server_key_iv: "636e77db4bb43d008d712430e98079b5"
server_key_version: "encryptionkey@0"
I20260619 16:45:30.211295 19535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:30.236145 19535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:30.239385 19535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:30.239614 19535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:30.247371 19535 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46113
I20260619 16:45:30.247385 19602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46113 every 8 connection(s)
I20260619 16:45:30.248463 19535 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:45:30.251551 19603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:30.256704 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 19535
I20260619 16:45:30.256870 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:45:30.257165 392 external_mini_cluster.cc:1428] Setting key 9ac9568dd1bb24bacf47330cab47ecaa
I20260619 16:45:30.258282 19603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab0c781ac6e54b84b4c9047d8cad0386: Bootstrap starting.
I20260619 16:45:30.261147 19603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab0c781ac6e54b84b4c9047d8cad0386: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:30.262032 19603 log.cc:824] T 00000000000000000000000000000000 P ab0c781ac6e54b84b4c9047d8cad0386: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:30.264469 19603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab0c781ac6e54b84b4c9047d8cad0386: No bootstrap required, opened a new log
Jun 19 16:45:30 dist-test-slave-03cd krb5kdc[19522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:45:30.268919 19603 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab0c781ac6e54b84b4c9047d8cad0386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab0c781ac6e54b84b4c9047d8cad0386" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } }
I20260619 16:45:30.269222 19603 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab0c781ac6e54b84b4c9047d8cad0386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:30.269309 19603 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab0c781ac6e54b84b4c9047d8cad0386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab0c781ac6e54b84b4c9047d8cad0386, State: Initialized, Role: FOLLOWER
I20260619 16:45:30.269780 19603 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab0c781ac6e54b84b4c9047d8cad0386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab0c781ac6e54b84b4c9047d8cad0386" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } }
I20260619 16:45:30.269935 19603 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab0c781ac6e54b84b4c9047d8cad0386 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:45:30.270022 19603 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab0c781ac6e54b84b4c9047d8cad0386 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:45:30.270153 19603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab0c781ac6e54b84b4c9047d8cad0386 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:30.271147 19603 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab0c781ac6e54b84b4c9047d8cad0386 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab0c781ac6e54b84b4c9047d8cad0386" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } }
I20260619 16:45:30.271515 19603 leader_election.cc:302] T 00000000000000000000000000000000 P ab0c781ac6e54b84b4c9047d8cad0386 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab0c781ac6e54b84b4c9047d8cad0386; no voters:
I20260619 16:45:30.271907 19603 leader_election.cc:288] T 00000000000000000000000000000000 P ab0c781ac6e54b84b4c9047d8cad0386 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:45:30.272032 19608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab0c781ac6e54b84b4c9047d8cad0386 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:30.272261 19608 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab0c781ac6e54b84b4c9047d8cad0386 [term 1 LEADER]: Becoming Leader. State: Replica: ab0c781ac6e54b84b4c9047d8cad0386, State: Running, Role: LEADER
I20260619 16:45:30.272650 19608 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab0c781ac6e54b84b4c9047d8cad0386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab0c781ac6e54b84b4c9047d8cad0386" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } }
I20260619 16:45:30.273442 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:30.258732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58030 (local address 127.0.98.62:46113)
0619 16:45:30.259182 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:30.259203 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:30.259252 (+ 49us) server_negotiation.cc:407] Connection header received
0619 16:45:30.259992 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:30.260011 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:30.260388 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:30.260684 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:30.261695 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:30.262810 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:30.264000 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:30.264510 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:30.264736 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:30.267455 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:30.267461 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:30.267478 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:30.267512 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:30.269723 (+ 2211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:30.270378 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:30.270384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:30.270391 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:30.270529 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:30.270862 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:30.270865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:30.270867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:30.271426 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:30.271647 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:30.272070 (+ 423us) server_negotiation.cc:299] Negotiation successful
0619 16:45:30.272457 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":149,"threads_started":1}
I20260619 16:45:30.273464 19603 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab0c781ac6e54b84b4c9047d8cad0386 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:45:30.274681 19610 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab0c781ac6e54b84b4c9047d8cad0386 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab0c781ac6e54b84b4c9047d8cad0386" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab0c781ac6e54b84b4c9047d8cad0386" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } } }
I20260619 16:45:30.274811 19610 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab0c781ac6e54b84b4c9047d8cad0386 [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:30.275295 19609 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab0c781ac6e54b84b4c9047d8cad0386 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab0c781ac6e54b84b4c9047d8cad0386. Latest consensus state: current_term: 1 leader_uuid: "ab0c781ac6e54b84b4c9047d8cad0386" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab0c781ac6e54b84b4c9047d8cad0386" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } } }
I20260619 16:45:30.276125 19609 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab0c781ac6e54b84b4c9047d8cad0386 [sys.catalog]: This master's current role is: LEADER
W20260619 16:45:30.278473 19623 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ab0c781ac6e54b84b4c9047d8cad0386: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:45:30.278581 19623 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:45:30.278659 19617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:45:30.279533 19617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:45:30.285974 19617 catalog_manager.cc:1347] Generated new cluster ID: cc639e07e72f468691a50e0c0fdb1df9
I20260619 16:45:30.286078 19617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:45:30.299016 19617 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:45:30.300097 19617 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:45:30.324558 19617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab0c781ac6e54b84b4c9047d8cad0386: Generated new TSK 0
I20260619 16:45:30.325349 19617 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:45:30.368645 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46113
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:30.490535 19629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:30.490864 19629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:30.490911 19629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:30.495580 19629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:30.495687 19629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:30.495800 19629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:45:30.501566 19629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:30.503051 19629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:30.504345 19629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:30.515357 19635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:30.515352 19637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:30.515357 19634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:30.515792 19629 server_base.cc:1034] running on GCE node
I20260619 16:45:30.516299 19629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:30.517021 19629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:30.518247 19629 hybrid_clock.cc:648] HybridClock initialized: now 1781887530518192 us; error 71 us; skew 500 ppm
Jun 19 16:45:30 dist-test-slave-03cd krb5kdc[19522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:30.521309 19629 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:45:30.522197 19629 webserver.cc:460] Webserver started at http://127.0.98.1:40641/ using document root <none> and password file <none>
I20260619 16:45:30.522737 19629 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:30.522794 19629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:30.522974 19629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:30.525233 19629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-0/data/instance:
uuid: "aef24130c6d7473d9073b324fd35d274"
format_stamp: "Formatted at 2026-06-19 16:45:30 on dist-test-slave-03cd"
server_key: "32be60bed426087912d51707406581f9"
server_key_iv: "8d54deb23b8e2a0d9847ab25ad86f4c7"
server_key_version: "encryptionkey@0"
I20260619 16:45:30.525897 19629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-0/wal/instance:
uuid: "aef24130c6d7473d9073b324fd35d274"
format_stamp: "Formatted at 2026-06-19 16:45:30 on dist-test-slave-03cd"
server_key: "32be60bed426087912d51707406581f9"
server_key_iv: "8d54deb23b8e2a0d9847ab25ad86f4c7"
server_key_version: "encryptionkey@0"
I20260619 16:45:30.529942 19629 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:45:30.532784 19644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:30.534343 19629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:45:30.534642 19629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "aef24130c6d7473d9073b324fd35d274"
format_stamp: "Formatted at 2026-06-19 16:45:30 on dist-test-slave-03cd"
server_key: "32be60bed426087912d51707406581f9"
server_key_iv: "8d54deb23b8e2a0d9847ab25ad86f4c7"
server_key_version: "encryptionkey@0"
I20260619 16:45:30.534807 19629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:30.549257 19629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:30.552719 19629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:30.552942 19629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:30.553544 19629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:45:30.554385 19629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:45:30.554432 19629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:30.554500 19629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:45:30.554533 19629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:30.565356 19629 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35531
I20260619 16:45:30.565376 19757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35531 every 8 connection(s)
I20260619 16:45:30.566494 19629 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 16:45:30 dist-test-slave-03cd krb5kdc[19522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:45:30.576838 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 19629
I20260619 16:45:30.576977 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:45:30.577334 392 external_mini_cluster.cc:1428] Setting key 18944a94fe0c225338ff3d2d6a4fabd3
I20260619 16:45:30.581900 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:30.568629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54627 (local address 127.0.98.62:46113)
0619 16:45:30.568821 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:30.568826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:30.569924 (+ 1098us) server_negotiation.cc:407] Connection header received
0619 16:45:30.570773 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:30.570776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:30.570825 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:30.570911 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:30.572167 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:30.572759 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:30.573930 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:30.574287 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:30.574389 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:30.577338 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:30.577345 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:30.577368 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:30.577407 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:30.579313 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:30.579916 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:30.579923 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:30.579928 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:30.580027 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:30.580400 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:30.580406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:30.580411 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:30.580691 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:30.580802 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:30.581624 (+ 822us) server_negotiation.cc:299] Negotiation successful
0619 16:45:30.581760 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 16:45:30.583101 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:30.569020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46113 (local address 127.0.98.1:54627)
0619 16:45:30.569753 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:30.569797 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:30.570574 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:30.571069 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:30.571075 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:30.571470 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:45:30.571987 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:30.572001 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:30.572879 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:30.572881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:30.573788 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:30.573798 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:30.574448 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:30.574451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:30.574601 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:30.575321 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:45:30.575338 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:45:30.577145 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:45:30.579441 (+ 2296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:45:30.579445 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:45:30.579461 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:45:30.579814 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:45:30.580123 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:45:30.580125 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:45:30.580126 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:45:30.580294 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:45:30.580817 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:45:30.580824 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:45:30.581207 (+ 383us) client_negotiation.cc:769] Sending connection context
0619 16:45:30.581507 (+ 300us) client_negotiation.cc:240] Negotiation successful
0619 16:45:30.581923 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":140,"threads_started":1}
I20260619 16:45:30.584705 19758 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46113
I20260619 16:45:30.585024 19758 heartbeater.cc:464] Registering TS with master...
I20260619 16:45:30.585603 19758 heartbeater.cc:511] Master 127.0.98.62:46113 requested a full tablet report, sending...
I20260619 16:45:30.587538 19567 ts_manager.cc:194] Registered new tserver with Master: aef24130c6d7473d9073b324fd35d274 (127.0.98.1:35531)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:45:30.588876 19567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:45:30.607952 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46113
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:30.726146 19763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:30.726501 19763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:30.726554 19763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:30.730993 19763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:30.731104 19763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:30.731174 19763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:45:30.737015 19763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:30.738176 19763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:30.739286 19763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:30.749995 19768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:30.749995 19769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:30.750036 19771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:30.750528 19763 server_base.cc:1034] running on GCE node
I20260619 16:45:30.751006 19763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:30.751812 19763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:30.753000 19763 hybrid_clock.cc:648] HybridClock initialized: now 1781887530752975 us; error 38 us; skew 500 ppm
Jun 19 16:45:30 dist-test-slave-03cd krb5kdc[19522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:30.756150 19763 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:45:30.757218 19763 webserver.cc:460] Webserver started at http://127.0.98.2:38549/ using document root <none> and password file <none>
I20260619 16:45:30.757863 19763 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:30.757961 19763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:30.758203 19763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:30.760488 19763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-1/data/instance:
uuid: "b0377f7833bc47b3ab69e0f6ef1db747"
format_stamp: "Formatted at 2026-06-19 16:45:30 on dist-test-slave-03cd"
server_key: "4e4c530eaea2a53138c5f46f075bf79c"
server_key_iv: "acd789f48117e8985b2858deee4eda77"
server_key_version: "encryptionkey@0"
I20260619 16:45:30.761094 19763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b0377f7833bc47b3ab69e0f6ef1db747"
format_stamp: "Formatted at 2026-06-19 16:45:30 on dist-test-slave-03cd"
server_key: "4e4c530eaea2a53138c5f46f075bf79c"
server_key_iv: "acd789f48117e8985b2858deee4eda77"
server_key_version: "encryptionkey@0"
I20260619 16:45:30.765617 19763 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:45:30.768401 19778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:30.769824 19763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:45:30.769989 19763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "b0377f7833bc47b3ab69e0f6ef1db747"
format_stamp: "Formatted at 2026-06-19 16:45:30 on dist-test-slave-03cd"
server_key: "4e4c530eaea2a53138c5f46f075bf79c"
server_key_iv: "acd789f48117e8985b2858deee4eda77"
server_key_version: "encryptionkey@0"
I20260619 16:45:30.770128 19763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:30.791458 19763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:30.795336 19763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:30.795624 19763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:30.796340 19763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:45:30.797245 19763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:45:30.797322 19763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:30.797433 19763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:45:30.797485 19763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:30.807351 19763 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40701
I20260619 16:45:30.807457 19891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40701 every 8 connection(s)
I20260619 16:45:30.808516 19763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:45:30.816170 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 19763
I20260619 16:45:30.816347 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:45:30.816677 392 external_mini_cluster.cc:1428] Setting key 6466792484888f1b12efde452d71ddb6
Jun 19 16:45:30 dist-test-slave-03cd krb5kdc[19522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:45:30.824000 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:30.810508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59159 (local address 127.0.98.62:46113)
0619 16:45:30.810690 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:30.810694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:30.811865 (+ 1171us) server_negotiation.cc:407] Connection header received
0619 16:45:30.812824 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:30.812828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:30.812888 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:30.813000 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:30.814250 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:30.814795 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:30.816195 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:30.816536 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:30.816635 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:30.819543 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:30.819546 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:30.819549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:30.819574 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:30.821484 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:30.822117 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:30.822124 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:30.822129 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:30.822233 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:30.822706 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:30.822709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:30.822710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:30.822914 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:30.823048 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:30.823642 (+ 594us) server_negotiation.cc:299] Negotiation successful
0619 16:45:30.823824 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 16:45:30.825191 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:30.810914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46113 (local address 127.0.98.2:59159)
0619 16:45:30.811620 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:30.811740 (+ 120us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:30.812606 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:30.813187 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:30.813194 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:30.813568 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:45:30.814062 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:30.814076 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:30.814969 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:30.814972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:30.816060 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:30.816073 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:30.816711 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:30.816718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:30.816934 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:30.817678 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:45:30.817699 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:45:30.819343 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:45:30.821653 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:45:30.821662 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:45:30.821682 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:45:30.821995 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:45:30.822358 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:45:30.822361 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:45:30.822363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:45:30.822528 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:45:30.823049 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:45:30.823058 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:45:30.823378 (+ 320us) client_negotiation.cc:769] Sending connection context
0619 16:45:30.823597 (+ 219us) client_negotiation.cc:240] Negotiation successful
0619 16:45:30.824053 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":124,"threads_started":1}
I20260619 16:45:30.826682 19892 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46113
I20260619 16:45:30.827013 19892 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:45:30.827749 19892 heartbeater.cc:511] Master 127.0.98.62:46113 requested a full tablet report, sending...
I20260619 16:45:30.829257 19567 ts_manager.cc:194] Registered new tserver with Master: b0377f7833bc47b3ab69e0f6ef1db747 (127.0.98.2:40701)
I20260619 16:45:30.829898 19567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59159
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:45:30.852823 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46113
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:30.968616 19897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:30.968957 19897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:30.969035 19897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:30.973670 19897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:30.973790 19897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:30.973886 19897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:45:30.979926 19897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:30.981128 19897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:30.982182 19897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:30.992298 19903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:30.992336 19902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:30.992362 19897 server_base.cc:1034] running on GCE node
W20260619 16:45:30.992640 19905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:30.993101 19897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:30.993849 19897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:30.995049 19897 hybrid_clock.cc:648] HybridClock initialized: now 1781887530995027 us; error 39 us; skew 500 ppm
Jun 19 16:45:30 dist-test-slave-03cd krb5kdc[19522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:30.998414 19897 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:45:30.999514 19897 webserver.cc:460] Webserver started at http://127.0.98.3:39705/ using document root <none> and password file <none>
I20260619 16:45:31.000214 19897 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:31.000322 19897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:31.000562 19897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:31.002743 19897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-2/data/instance:
uuid: "64591153cbe1401094feb1528b2c55fd"
format_stamp: "Formatted at 2026-06-19 16:45:30 on dist-test-slave-03cd"
server_key: "a88ed08dcddb2d622c6034e77fc0db6c"
server_key_iv: "be727f0280c18d4a17a19a35e13430e4"
server_key_version: "encryptionkey@0"
I20260619 16:45:31.003362 19897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-2/wal/instance:
uuid: "64591153cbe1401094feb1528b2c55fd"
format_stamp: "Formatted at 2026-06-19 16:45:30 on dist-test-slave-03cd"
server_key: "a88ed08dcddb2d622c6034e77fc0db6c"
server_key_iv: "be727f0280c18d4a17a19a35e13430e4"
server_key_version: "encryptionkey@0"
I20260619 16:45:31.007616 19897 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260619 16:45:31.010845 19912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:31.012603 19897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:45:31.012789 19897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "64591153cbe1401094feb1528b2c55fd"
format_stamp: "Formatted at 2026-06-19 16:45:30 on dist-test-slave-03cd"
server_key: "a88ed08dcddb2d622c6034e77fc0db6c"
server_key_iv: "be727f0280c18d4a17a19a35e13430e4"
server_key_version: "encryptionkey@0"
I20260619 16:45:31.012943 19897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:31.029791 19897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:31.033339 19897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:31.033563 19897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:31.034204 19897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:45:31.035065 19897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:45:31.035137 19897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:31.035208 19897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:45:31.035260 19897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:31.045493 19897 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39453
I20260619 16:45:31.045516 20025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39453 every 8 connection(s)
I20260619 16:45:31.046557 19897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:45:31.049804 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 19897
I20260619 16:45:31.049954 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:45:31.050225 392 external_mini_cluster.cc:1428] Setting key 82a4faa7e7f10748064a1ecd55eaf146
Jun 19 16:45:31 dist-test-slave-03cd krb5kdc[19522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:45:31.061033 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:31.048495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59089 (local address 127.0.98.62:46113)
0619 16:45:31.048644 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:31.048648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:31.049750 (+ 1102us) server_negotiation.cc:407] Connection header received
0619 16:45:31.050767 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:31.050771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:31.050829 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:31.050901 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:31.052194 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.052728 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.053950 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.054176 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.054303 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:31.057030 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:31.057033 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:31.057035 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:31.057063 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:31.058551 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:31.059160 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:31.059164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:31.059166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:31.059294 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:31.059790 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:31.059794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:31.059796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:31.060011 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:31.060130 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:31.060705 (+ 575us) server_negotiation.cc:299] Negotiation successful
0619 16:45:31.060853 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:45:31.061928 20028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:31.048907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46113 (local address 127.0.98.3:59089)
0619 16:45:31.049580 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:31.049623 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:31.050555 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:31.051077 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:31.051084 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:31.051456 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:45:31.052040 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:31.052055 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.052871 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.052875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:31.053806 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:31.053816 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.054351 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.054355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:31.054546 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:31.055194 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:45:31.055214 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:45:31.056826 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:45:31.058698 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:45:31.058705 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:45:31.058725 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:45:31.059042 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:45:31.059460 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:45:31.059463 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:45:31.059466 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:45:31.059635 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:45:31.060140 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:45:31.060149 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:45:31.060445 (+ 296us) client_negotiation.cc:769] Sending connection context
0619 16:45:31.060645 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 16:45:31.061025 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":116,"threads_started":1}
I20260619 16:45:31.063396 20026 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46113
I20260619 16:45:31.063779 20026 heartbeater.cc:464] Registering TS with master...
I20260619 16:45:31.064455 20026 heartbeater.cc:511] Master 127.0.98.62:46113 requested a full tablet report, sending...
I20260619 16:45:31.066008 19567 ts_manager.cc:194] Registered new tserver with Master: 64591153cbe1401094feb1528b2c55fd (127.0.98.3:39453)
I20260619 16:45:31.067013 19567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59089
I20260619 16:45:31.078334 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:45:31.090839 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:31.081042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58046 (local address 127.0.98.62:46113)
0619 16:45:31.081216 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:31.081221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:31.081410 (+ 189us) server_negotiation.cc:407] Connection header received
0619 16:45:31.081626 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:31.081630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:31.081697 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:31.081862 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:31.082800 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.083477 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.084594 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.084985 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.085095 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:31.086931 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:31.086935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:31.086938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:31.086977 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:31.088670 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:31.089247 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:31.089250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:31.089252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:31.089354 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:31.089686 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:31.089691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:31.089694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:31.090100 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:31.090237 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:31.090518 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 16:45:31.090651 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 16:45:31.094878 19567 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:45:31.098869 19567 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:45:31.119253 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:31.110083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35531 (local address 127.0.0.1:42336)
0619 16:45:31.110490 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:31.110543 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:31.110676 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:31.112759 (+ 2083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:31.112767 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:31.112797 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:31.112967 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:31.112975 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.114559 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.114562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:31.117074 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:31.117085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.118609 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.118613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:31.118782 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:31.118825 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 16:45:31.118933 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 16:45:31.119083 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":1151,"thread_start_us":129,"threads_started":1}
I20260619 16:45:31.119251 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:31.111982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42336 (local address 127.0.98.1:35531)
0619 16:45:31.112263 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:31.112271 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:31.112295 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:45:31.112367 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:31.112371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:31.112549 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:31.113136 (+ 587us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:31.113283 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.114429 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.117651 (+ 3222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.118466 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.118770 (+ 304us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:31.118913 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:31.118981 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 16:45:31.119083 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":122,"threads_started":1}
I20260619 16:45:31.119536 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:31.110992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39453 (local address 127.0.0.1:44968)
0619 16:45:31.111323 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:31.111339 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:31.111439 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:31.113330 (+ 1891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:31.113333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:31.113349 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:31.113433 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:31.113439 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.114530 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.114534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:31.116432 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:31.116446 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.118984 (+ 2538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.118990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:31.119119 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:31.119157 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 16:45:31.119217 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 16:45:31.119329 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":3,"thread_start_us":139,"threads_started":1}
I20260619 16:45:31.120074 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:31.112524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40701 (local address 127.0.0.1:39474)
0619 16:45:31.113648 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:31.113662 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:31.113747 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:31.114440 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:31.114443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:31.114463 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:31.114596 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:31.114602 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.116136 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.116139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:31.117270 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:31.117280 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.118619 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.118622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:31.119736 (+ 1114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:31.119795 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 16:45:31.119865 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 16:45:31.119940 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1061,"mutex_wait_us":30,"thread_start_us":110,"threads_started":1}
I20260619 16:45:31.120517 20039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:31.111142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44968 (local address 127.0.98.3:39453)
0619 16:45:31.112072 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:31.112091 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:31.112119 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:45:31.112213 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:31.112219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:31.112475 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:31.112652 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:31.113545 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.114374 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.116610 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.118820 (+ 2210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.119918 (+ 1098us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:31.120103 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:31.120166 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 16:45:31.120309 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":785,"thread_start_us":143,"threads_started":1}
I20260619 16:45:31.120600 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:31.112729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39474 (local address 127.0.98.2:40701)
0619 16:45:31.113067 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:31.113076 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:31.113843 (+ 767us) server_negotiation.cc:407] Connection header received
0619 16:45:31.113924 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:31.113929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:31.114132 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:31.114280 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:31.114724 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.115959 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.117444 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.118281 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.118393 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:31.118568 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:31.120330 (+ 1762us) server_negotiation.cc:299] Negotiation successful
0619 16:45:31.120452 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":153,"threads_started":1}
I20260619 16:45:31.123292 19692 tablet_service.cc:1467] Processing CreateTablet for tablet 8107838b0d294ac8b78386bd44dc85a7 (DEFAULT_TABLE table=test-table [id=3d7559e3cf1e47408ad061ac11b36791]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:45:31.123304 19826 tablet_service.cc:1467] Processing CreateTablet for tablet 8107838b0d294ac8b78386bd44dc85a7 (DEFAULT_TABLE table=test-table [id=3d7559e3cf1e47408ad061ac11b36791]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:45:31.124567 19692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8107838b0d294ac8b78386bd44dc85a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:31.124567 19826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8107838b0d294ac8b78386bd44dc85a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:31.130504 19960 tablet_service.cc:1467] Processing CreateTablet for tablet 8107838b0d294ac8b78386bd44dc85a7 (DEFAULT_TABLE table=test-table [id=3d7559e3cf1e47408ad061ac11b36791]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:45:31.131750 19960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8107838b0d294ac8b78386bd44dc85a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:31.132563 20044 tablet_bootstrap.cc:492] T 8107838b0d294ac8b78386bd44dc85a7 P b0377f7833bc47b3ab69e0f6ef1db747: Bootstrap starting.
I20260619 16:45:31.132968 20043 tablet_bootstrap.cc:492] T 8107838b0d294ac8b78386bd44dc85a7 P aef24130c6d7473d9073b324fd35d274: Bootstrap starting.
I20260619 16:45:31.135617 20043 tablet_bootstrap.cc:654] T 8107838b0d294ac8b78386bd44dc85a7 P aef24130c6d7473d9073b324fd35d274: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:31.138314 20044 tablet_bootstrap.cc:654] T 8107838b0d294ac8b78386bd44dc85a7 P b0377f7833bc47b3ab69e0f6ef1db747: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:31.139739 20044 log.cc:824] T 8107838b0d294ac8b78386bd44dc85a7 P b0377f7833bc47b3ab69e0f6ef1db747: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:31.140825 20046 tablet_bootstrap.cc:492] T 8107838b0d294ac8b78386bd44dc85a7 P 64591153cbe1401094feb1528b2c55fd: Bootstrap starting.
I20260619 16:45:31.141098 20043 log.cc:824] T 8107838b0d294ac8b78386bd44dc85a7 P aef24130c6d7473d9073b324fd35d274: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:31.142762 20044 tablet_bootstrap.cc:492] T 8107838b0d294ac8b78386bd44dc85a7 P b0377f7833bc47b3ab69e0f6ef1db747: No bootstrap required, opened a new log
I20260619 16:45:31.142987 20044 ts_tablet_manager.cc:1397] T 8107838b0d294ac8b78386bd44dc85a7 P b0377f7833bc47b3ab69e0f6ef1db747: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260619 16:45:31.143236 20043 tablet_bootstrap.cc:492] T 8107838b0d294ac8b78386bd44dc85a7 P aef24130c6d7473d9073b324fd35d274: No bootstrap required, opened a new log
I20260619 16:45:31.143394 20046 tablet_bootstrap.cc:654] T 8107838b0d294ac8b78386bd44dc85a7 P 64591153cbe1401094feb1528b2c55fd: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:31.143441 20043 ts_tablet_manager.cc:1397] T 8107838b0d294ac8b78386bd44dc85a7 P aef24130c6d7473d9073b324fd35d274: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260619 16:45:31.144707 20046 log.cc:824] T 8107838b0d294ac8b78386bd44dc85a7 P 64591153cbe1401094feb1528b2c55fd: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:31.147001 20046 tablet_bootstrap.cc:492] T 8107838b0d294ac8b78386bd44dc85a7 P 64591153cbe1401094feb1528b2c55fd: No bootstrap required, opened a new log
I20260619 16:45:31.147212 20046 ts_tablet_manager.cc:1397] T 8107838b0d294ac8b78386bd44dc85a7 P 64591153cbe1401094feb1528b2c55fd: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 16:45:31.147213 20044 raft_consensus.cc:348] T 8107838b0d294ac8b78386bd44dc85a7 P b0377f7833bc47b3ab69e0f6ef1db747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aef24130c6d7473d9073b324fd35d274" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35531 } } peers { permanent_uuid: "64591153cbe1401094feb1528b2c55fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39453 } } peers { permanent_uuid: "b0377f7833bc47b3ab69e0f6ef1db747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40701 } }
I20260619 16:45:31.147315 20043 raft_consensus.cc:348] T 8107838b0d294ac8b78386bd44dc85a7 P aef24130c6d7473d9073b324fd35d274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aef24130c6d7473d9073b324fd35d274" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35531 } } peers { permanent_uuid: "64591153cbe1401094feb1528b2c55fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39453 } } peers { permanent_uuid: "b0377f7833bc47b3ab69e0f6ef1db747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40701 } }
I20260619 16:45:31.147529 20044 raft_consensus.cc:374] T 8107838b0d294ac8b78386bd44dc85a7 P b0377f7833bc47b3ab69e0f6ef1db747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:31.147579 20043 raft_consensus.cc:374] T 8107838b0d294ac8b78386bd44dc85a7 P aef24130c6d7473d9073b324fd35d274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:31.147583 20044 raft_consensus.cc:729] T 8107838b0d294ac8b78386bd44dc85a7 P b0377f7833bc47b3ab69e0f6ef1db747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0377f7833bc47b3ab69e0f6ef1db747, State: Initialized, Role: FOLLOWER
I20260619 16:45:31.147626 20043 raft_consensus.cc:729] T 8107838b0d294ac8b78386bd44dc85a7 P aef24130c6d7473d9073b324fd35d274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aef24130c6d7473d9073b324fd35d274, State: Initialized, Role: FOLLOWER
I20260619 16:45:31.148118 20043 consensus_queue.cc:260] T 8107838b0d294ac8b78386bd44dc85a7 P aef24130c6d7473d9073b324fd35d274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aef24130c6d7473d9073b324fd35d274" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35531 } } peers { permanent_uuid: "64591153cbe1401094feb1528b2c55fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39453 } } peers { permanent_uuid: "b0377f7833bc47b3ab69e0f6ef1db747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40701 } }
I20260619 16:45:31.148115 20044 consensus_queue.cc:260] T 8107838b0d294ac8b78386bd44dc85a7 P b0377f7833bc47b3ab69e0f6ef1db747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aef24130c6d7473d9073b324fd35d274" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35531 } } peers { permanent_uuid: "64591153cbe1401094feb1528b2c55fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39453 } } peers { permanent_uuid: "b0377f7833bc47b3ab69e0f6ef1db747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40701 } }
I20260619 16:45:31.149188 20043 ts_tablet_manager.cc:1428] T 8107838b0d294ac8b78386bd44dc85a7 P aef24130c6d7473d9073b324fd35d274: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260619 16:45:31.149201 20044 ts_tablet_manager.cc:1428] T 8107838b0d294ac8b78386bd44dc85a7 P b0377f7833bc47b3ab69e0f6ef1db747: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 16:45:31.149564 19758 heartbeater.cc:503] Master 127.0.98.62:46113 was elected leader, sending a full tablet report...
I20260619 16:45:31.149564 19892 heartbeater.cc:503] Master 127.0.98.62:46113 was elected leader, sending a full tablet report...
I20260619 16:45:31.152173 20046 raft_consensus.cc:348] T 8107838b0d294ac8b78386bd44dc85a7 P 64591153cbe1401094feb1528b2c55fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aef24130c6d7473d9073b324fd35d274" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35531 } } peers { permanent_uuid: "64591153cbe1401094feb1528b2c55fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39453 } } peers { permanent_uuid: "b0377f7833bc47b3ab69e0f6ef1db747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40701 } }
I20260619 16:45:31.152479 20046 raft_consensus.cc:374] T 8107838b0d294ac8b78386bd44dc85a7 P 64591153cbe1401094feb1528b2c55fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:31.152545 20046 raft_consensus.cc:729] T 8107838b0d294ac8b78386bd44dc85a7 P 64591153cbe1401094feb1528b2c55fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64591153cbe1401094feb1528b2c55fd, State: Initialized, Role: FOLLOWER
I20260619 16:45:31.153069 20046 consensus_queue.cc:260] T 8107838b0d294ac8b78386bd44dc85a7 P 64591153cbe1401094feb1528b2c55fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aef24130c6d7473d9073b324fd35d274" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35531 } } peers { permanent_uuid: "64591153cbe1401094feb1528b2c55fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39453 } } peers { permanent_uuid: "b0377f7833bc47b3ab69e0f6ef1db747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40701 } }
I20260619 16:45:31.153810 20026 heartbeater.cc:503] Master 127.0.98.62:46113 was elected leader, sending a full tablet report...
I20260619 16:45:31.155267 20046 ts_tablet_manager.cc:1428] T 8107838b0d294ac8b78386bd44dc85a7 P 64591153cbe1401094feb1528b2c55fd: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260619 16:45:31.297622 20027 tablet.cc:2369] T 8107838b0d294ac8b78386bd44dc85a7 P 64591153cbe1401094feb1528b2c55fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:45:31.309863 19893 tablet.cc:2369] T 8107838b0d294ac8b78386bd44dc85a7 P b0377f7833bc47b3ab69e0f6ef1db747: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:45:31.318107 19759 tablet.cc:2369] T 8107838b0d294ac8b78386bd44dc85a7 P aef24130c6d7473d9073b324fd35d274: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:45:31.331635 20049 raft_consensus.cc:482] T 8107838b0d294ac8b78386bd44dc85a7 P aef24130c6d7473d9073b324fd35d274 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:45:31.331993 20049 raft_consensus.cc:504] T 8107838b0d294ac8b78386bd44dc85a7 P aef24130c6d7473d9073b324fd35d274 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aef24130c6d7473d9073b324fd35d274" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35531 } } peers { permanent_uuid: "64591153cbe1401094feb1528b2c55fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39453 } } peers { permanent_uuid: "b0377f7833bc47b3ab69e0f6ef1db747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40701 } }
I20260619 16:45:31.333238 20049 leader_election.cc:288] T 8107838b0d294ac8b78386bd44dc85a7 P aef24130c6d7473d9073b324fd35d274 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64591153cbe1401094feb1528b2c55fd (127.0.98.3:39453), b0377f7833bc47b3ab69e0f6ef1db747 (127.0.98.2:40701)
I20260619 16:45:31.337450 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:31.333667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35445 (local address 127.0.98.2:40701)
0619 16:45:31.333833 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:31.333839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:31.334144 (+ 305us) server_negotiation.cc:407] Connection header received
0619 16:45:31.334214 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:31.334216 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:31.334266 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:31.334400 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:31.334752 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.335249 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.336433 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.336839 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.336965 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:31.337032 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:31.337212 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 16:45:31.337290 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 16:45:31.337471 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:31.333572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40701 (local address 127.0.98.1:35445)
0619 16:45:31.334011 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:31.334023 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:31.334136 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:31.334416 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:31.334418 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:31.334442 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:31.334582 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:31.334587 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.335371 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.335376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:31.336301 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:31.336311 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.336973 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.336977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:31.337084 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:31.337105 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 16:45:31.337194 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 16:45:31.337283 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":1,"thread_start_us":150,"threads_started":1}
I20260619 16:45:31.337604 20039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:31.333672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46035 (local address 127.0.98.3:39453)
0619 16:45:31.333842 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:31.333847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:31.334429 (+ 582us) server_negotiation.cc:407] Connection header received
0619 16:45:31.334589 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:31.334592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:31.334644 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:31.334725 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:31.334990 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.335474 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.336567 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.336984 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.337108 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:31.337172 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:31.337372 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 16:45:31.337448 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:45:31.337872 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:31.333572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39453 (local address 127.0.98.1:46035)
0619 16:45:31.334325 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:31.334334 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:31.334470 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:31.334747 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:31.334750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:31.334765 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:31.334877 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:31.334883 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.335587 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.335592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:31.336436 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:31.336445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.337109 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.337113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:31.337227 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:31.337255 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 16:45:31.337358 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 16:45:31.337434 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":39,"spinlock_wait_cycles":33024,"thread_start_us":115,"threads_started":1}
I20260619 16:45:31.338282 19846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8107838b0d294ac8b78386bd44dc85a7" candidate_uuid: "aef24130c6d7473d9073b324fd35d274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0377f7833bc47b3ab69e0f6ef1db747" is_pre_election: true
I20260619 16:45:31.338579 19980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8107838b0d294ac8b78386bd44dc85a7" candidate_uuid: "aef24130c6d7473d9073b324fd35d274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64591153cbe1401094feb1528b2c55fd" is_pre_election: true
I20260619 16:45:31.338649 19846 raft_consensus.cc:2454] T 8107838b0d294ac8b78386bd44dc85a7 P b0377f7833bc47b3ab69e0f6ef1db747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aef24130c6d7473d9073b324fd35d274 in term 0.
I20260619 16:45:31.338907 19980 raft_consensus.cc:2454] T 8107838b0d294ac8b78386bd44dc85a7 P 64591153cbe1401094feb1528b2c55fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aef24130c6d7473d9073b324fd35d274 in term 0.
I20260619 16:45:31.339231 19646 leader_election.cc:302] T 8107838b0d294ac8b78386bd44dc85a7 P aef24130c6d7473d9073b324fd35d274 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aef24130c6d7473d9073b324fd35d274, b0377f7833bc47b3ab69e0f6ef1db747; no voters:
I20260619 16:45:31.339645 20049 raft_consensus.cc:2781] T 8107838b0d294ac8b78386bd44dc85a7 P aef24130c6d7473d9073b324fd35d274 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:45:31.339769 20049 raft_consensus.cc:482] T 8107838b0d294ac8b78386bd44dc85a7 P aef24130c6d7473d9073b324fd35d274 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:45:31.339821 20049 raft_consensus.cc:3037] T 8107838b0d294ac8b78386bd44dc85a7 P aef24130c6d7473d9073b324fd35d274 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:31.340883 20049 raft_consensus.cc:504] T 8107838b0d294ac8b78386bd44dc85a7 P aef24130c6d7473d9073b324fd35d274 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aef24130c6d7473d9073b324fd35d274" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35531 } } peers { permanent_uuid: "64591153cbe1401094feb1528b2c55fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39453 } } peers { permanent_uuid: "b0377f7833bc47b3ab69e0f6ef1db747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40701 } }
I20260619 16:45:31.341264 20049 leader_election.cc:288] T 8107838b0d294ac8b78386bd44dc85a7 P aef24130c6d7473d9073b324fd35d274 [CANDIDATE]: Term 1 election: Requested vote from peers 64591153cbe1401094feb1528b2c55fd (127.0.98.3:39453), b0377f7833bc47b3ab69e0f6ef1db747 (127.0.98.2:40701)
I20260619 16:45:31.341763 19980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8107838b0d294ac8b78386bd44dc85a7" candidate_uuid: "aef24130c6d7473d9073b324fd35d274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64591153cbe1401094feb1528b2c55fd"
I20260619 16:45:31.341799 19846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8107838b0d294ac8b78386bd44dc85a7" candidate_uuid: "aef24130c6d7473d9073b324fd35d274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0377f7833bc47b3ab69e0f6ef1db747"
I20260619 16:45:31.341929 19846 raft_consensus.cc:3037] T 8107838b0d294ac8b78386bd44dc85a7 P b0377f7833bc47b3ab69e0f6ef1db747 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:31.341919 19980 raft_consensus.cc:3037] T 8107838b0d294ac8b78386bd44dc85a7 P 64591153cbe1401094feb1528b2c55fd [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:31.343309 19846 raft_consensus.cc:2454] T 8107838b0d294ac8b78386bd44dc85a7 P b0377f7833bc47b3ab69e0f6ef1db747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aef24130c6d7473d9073b324fd35d274 in term 1.
I20260619 16:45:31.343309 19980 raft_consensus.cc:2454] T 8107838b0d294ac8b78386bd44dc85a7 P 64591153cbe1401094feb1528b2c55fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aef24130c6d7473d9073b324fd35d274 in term 1.
I20260619 16:45:31.343688 19646 leader_election.cc:302] T 8107838b0d294ac8b78386bd44dc85a7 P aef24130c6d7473d9073b324fd35d274 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aef24130c6d7473d9073b324fd35d274, b0377f7833bc47b3ab69e0f6ef1db747; no voters:
I20260619 16:45:31.343890 20049 raft_consensus.cc:2781] T 8107838b0d294ac8b78386bd44dc85a7 P aef24130c6d7473d9073b324fd35d274 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:31.344141 20049 raft_consensus.cc:686] T 8107838b0d294ac8b78386bd44dc85a7 P aef24130c6d7473d9073b324fd35d274 [term 1 LEADER]: Becoming Leader. State: Replica: aef24130c6d7473d9073b324fd35d274, State: Running, Role: LEADER
I20260619 16:45:31.344496 20049 consensus_queue.cc:237] T 8107838b0d294ac8b78386bd44dc85a7 P aef24130c6d7473d9073b324fd35d274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aef24130c6d7473d9073b324fd35d274" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35531 } } peers { permanent_uuid: "64591153cbe1401094feb1528b2c55fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39453 } } peers { permanent_uuid: "b0377f7833bc47b3ab69e0f6ef1db747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40701 } }
I20260619 16:45:31.348196 19565 catalog_manager.cc:5495] T 8107838b0d294ac8b78386bd44dc85a7 P aef24130c6d7473d9073b324fd35d274 reported cstate change: term changed from 0 to 1, leader changed from <none> to aef24130c6d7473d9073b324fd35d274 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "aef24130c6d7473d9073b324fd35d274" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aef24130c6d7473d9073b324fd35d274" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64591153cbe1401094feb1528b2c55fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0377f7833bc47b3ab69e0f6ef1db747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40701 } health_report { overall_health: UNKNOWN } } }
I20260619 16:45:31.364745 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:31.360710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42352 (local address 127.0.98.1:35531)
0619 16:45:31.360866 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:31.360871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:31.360885 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:45:31.361081 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:31.361083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:31.361144 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:31.361320 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:45:31.361565 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.362078 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.363004 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.363262 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.363408 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:31.363796 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:45:31.364273 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:45:31.364416 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:31.364479 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 16:45:31.364548 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:45:31.371467 19846 raft_consensus.cc:1261] T 8107838b0d294ac8b78386bd44dc85a7 P b0377f7833bc47b3ab69e0f6ef1db747 [term 1 FOLLOWER]: Refusing update from remote peer aef24130c6d7473d9073b324fd35d274: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:45:31.371520 19980 raft_consensus.cc:1261] T 8107838b0d294ac8b78386bd44dc85a7 P 64591153cbe1401094feb1528b2c55fd [term 1 FOLLOWER]: Refusing update from remote peer aef24130c6d7473d9073b324fd35d274: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:45:31.372375 20049 consensus_queue.cc:1035] T 8107838b0d294ac8b78386bd44dc85a7 P aef24130c6d7473d9073b324fd35d274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0377f7833bc47b3ab69e0f6ef1db747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:45:31.372614 20054 consensus_queue.cc:1035] T 8107838b0d294ac8b78386bd44dc85a7 P aef24130c6d7473d9073b324fd35d274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64591153cbe1401094feb1528b2c55fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:45:31.382081 20059 mvcc.cc:205] Tried to move back new op lower bound from 7298611328488894464 to 7298611328389955584. Current Snapshot: MvccSnapshot[applied={T|T < 7298611328488894464}]
I20260619 16:45:31.392196 19566 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:45:31.392418 19566 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:45:31.395511 19566 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab0c781ac6e54b84b4c9047d8cad0386: Sending DeleteTablet for 3 replicas of tablet 8107838b0d294ac8b78386bd44dc85a7
I20260619 16:45:31.396648 19826 tablet_service.cc:1514] Processing DeleteTablet for tablet 8107838b0d294ac8b78386bd44dc85a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:45:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39474
I20260619 16:45:31.396647 19692 tablet_service.cc:1514] Processing DeleteTablet for tablet 8107838b0d294ac8b78386bd44dc85a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:45:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42336
I20260619 16:45:31.396716 19960 tablet_service.cc:1514] Processing DeleteTablet for tablet 8107838b0d294ac8b78386bd44dc85a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:45:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44968
I20260619 16:45:31.397276 20069 tablet_replica.cc:330] T 8107838b0d294ac8b78386bd44dc85a7 P b0377f7833bc47b3ab69e0f6ef1db747: stopping tablet replica
I20260619 16:45:31.397279 20068 tablet_replica.cc:330] T 8107838b0d294ac8b78386bd44dc85a7 P 64591153cbe1401094feb1528b2c55fd: stopping tablet replica
I20260619 16:45:31.397677 20068 raft_consensus.cc:2229] T 8107838b0d294ac8b78386bd44dc85a7 P 64591153cbe1401094feb1528b2c55fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:45:31.397677 20069 raft_consensus.cc:2229] T 8107838b0d294ac8b78386bd44dc85a7 P b0377f7833bc47b3ab69e0f6ef1db747 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:45:31.398054 20069 raft_consensus.cc:2258] T 8107838b0d294ac8b78386bd44dc85a7 P b0377f7833bc47b3ab69e0f6ef1db747 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:45:31.398054 20068 raft_consensus.cc:2258] T 8107838b0d294ac8b78386bd44dc85a7 P 64591153cbe1401094feb1528b2c55fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:45:31.398433 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 19629
I20260619 16:45:31.400787 20068 ts_tablet_manager.cc:1905] T 8107838b0d294ac8b78386bd44dc85a7 P 64591153cbe1401094feb1528b2c55fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:45:31.400846 20069 ts_tablet_manager.cc:1905] T 8107838b0d294ac8b78386bd44dc85a7 P b0377f7833bc47b3ab69e0f6ef1db747: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:45:31.403798 20068 ts_tablet_manager.cc:1918] T 8107838b0d294ac8b78386bd44dc85a7 P 64591153cbe1401094feb1528b2c55fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:45:31.403798 20069 ts_tablet_manager.cc:1918] T 8107838b0d294ac8b78386bd44dc85a7 P b0377f7833bc47b3ab69e0f6ef1db747: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:45:31.403931 20068 log.cc:1196] T 8107838b0d294ac8b78386bd44dc85a7 P 64591153cbe1401094feb1528b2c55fd: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-2/wal/wals/8107838b0d294ac8b78386bd44dc85a7
I20260619 16:45:31.403931 20069 log.cc:1196] T 8107838b0d294ac8b78386bd44dc85a7 P b0377f7833bc47b3ab69e0f6ef1db747: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887259574980-392-0/minicluster-data/ts-1/wal/wals/8107838b0d294ac8b78386bd44dc85a7
I20260619 16:45:31.404280 20069 ts_tablet_manager.cc:1939] T 8107838b0d294ac8b78386bd44dc85a7 P b0377f7833bc47b3ab69e0f6ef1db747: Deleting consensus metadata
I20260619 16:45:31.404280 20068 ts_tablet_manager.cc:1939] T 8107838b0d294ac8b78386bd44dc85a7 P 64591153cbe1401094feb1528b2c55fd: Deleting consensus metadata
I20260619 16:45:31.405889 19554 catalog_manager.cc:4841] TS 64591153cbe1401094feb1528b2c55fd (127.0.98.3:39453): tablet 8107838b0d294ac8b78386bd44dc85a7 (table test-table [id=3d7559e3cf1e47408ad061ac11b36791]) successfully deleted
W20260619 16:45:31.406062 19553 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35531 (error 108)
I20260619 16:45:31.406126 19552 catalog_manager.cc:4841] TS b0377f7833bc47b3ab69e0f6ef1db747 (127.0.98.2:40701): tablet 8107838b0d294ac8b78386bd44dc85a7 (table test-table [id=3d7559e3cf1e47408ad061ac11b36791]) successfully deleted
I20260619 16:45:31.406920 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 19763
I20260619 16:45:31.407182 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:31.406864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35531 (local address 127.0.0.1:42358)
0619 16:45:31.406999 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:31.407079 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 16:45:31.407544 19553 catalog_manager.cc:4584] TS aef24130c6d7473d9073b324fd35d274 (127.0.98.1:35531): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8107838b0d294ac8b78386bd44dc85a7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35531: connect: Connection refused (error 111)
I20260619 16:45:31.412922 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 19897
I20260619 16:45:31.419322 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 19535
2026-06-19T16:45:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2581 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T16:45:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:45:31Z Disabled control of system clock
I20260619 16:45:31.451454 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43829
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:45:31.567284 20074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:31.567605 20074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:31.567641 20074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:31.572449 20074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:45:31.572547 20074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:31.572563 20074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:31.572582 20074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:45:31.572597 20074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:45:31.578408 20074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43829
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:31.579587 20074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:31.581012 20074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:31.590050 20080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:31.590054 20082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:31.590054 20079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:31.590472 20074 server_base.cc:1034] running on GCE node
I20260619 16:45:31.591084 20074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:31.592401 20074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:31.593592 20074 hybrid_clock.cc:648] HybridClock initialized: now 1781887531593566 us; error 41 us; skew 500 ppm
I20260619 16:45:31.600800 20074 webserver.cc:460] Webserver started at http://127.0.98.62:46491/ using document root <none> and password file <none>
I20260619 16:45:31.601487 20074 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:31.601560 20074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:31.601806 20074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:31.604094 20074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "7de07c72eebb45bfbb44e77101c50f5e"
format_stamp: "Formatted at 2026-06-19 16:45:31 on dist-test-slave-03cd"
server_key: "893ccf7c97942d3daef9b51550aea584"
server_key_iv: "b0c8d34e60995a480bc56a50cc7fa338"
server_key_version: "encryptionkey@0"
I20260619 16:45:31.604699 20074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "7de07c72eebb45bfbb44e77101c50f5e"
format_stamp: "Formatted at 2026-06-19 16:45:31 on dist-test-slave-03cd"
server_key: "893ccf7c97942d3daef9b51550aea584"
server_key_iv: "b0c8d34e60995a480bc56a50cc7fa338"
server_key_version: "encryptionkey@0"
I20260619 16:45:31.608665 20074 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:45:31.611362 20088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:31.612814 20074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:45:31.612998 20074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "7de07c72eebb45bfbb44e77101c50f5e"
format_stamp: "Formatted at 2026-06-19 16:45:31 on dist-test-slave-03cd"
server_key: "893ccf7c97942d3daef9b51550aea584"
server_key_iv: "b0c8d34e60995a480bc56a50cc7fa338"
server_key_version: "encryptionkey@0"
I20260619 16:45:31.613111 20074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:31.635617 20074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:31.636430 20074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:31.636639 20074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:31.644357 20074 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43829
I20260619 16:45:31.644368 20140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43829 every 8 connection(s)
I20260619 16:45:31.645341 20074 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:45:31.648136 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 20074
I20260619 16:45:31.648304 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:45:31.648593 392 external_mini_cluster.cc:1428] Setting key a316e556bdbe071784d39f3f7a848fae
I20260619 16:45:31.648773 20141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:31.655367 20141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7de07c72eebb45bfbb44e77101c50f5e: Bootstrap starting.
I20260619 16:45:31.657629 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:31.649922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44512 (local address 127.0.98.62:43829)
0619 16:45:31.650512 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:31.650527 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:31.650563 (+ 36us) server_negotiation.cc:407] Connection header received
0619 16:45:31.651213 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:31.651232 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:31.651612 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:31.652014 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:31.652433 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.653375 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.654258 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.654711 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.654899 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:31.655902 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:31.655907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:31.655919 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:31.656301 (+ 382us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:31.656374 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:31.656394 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:31.656504 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:31.656618 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 16:45:31.656921 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":106,"threads_started":1}
I20260619 16:45:31.657732 20141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7de07c72eebb45bfbb44e77101c50f5e: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:31.658618 20141 log.cc:824] T 00000000000000000000000000000000 P 7de07c72eebb45bfbb44e77101c50f5e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:31.660647 20141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7de07c72eebb45bfbb44e77101c50f5e: No bootstrap required, opened a new log
I20260619 16:45:31.664113 20141 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7de07c72eebb45bfbb44e77101c50f5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de07c72eebb45bfbb44e77101c50f5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43829 } }
I20260619 16:45:31.664358 20141 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7de07c72eebb45bfbb44e77101c50f5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:31.664434 20141 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7de07c72eebb45bfbb44e77101c50f5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7de07c72eebb45bfbb44e77101c50f5e, State: Initialized, Role: FOLLOWER
I20260619 16:45:31.664870 20141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7de07c72eebb45bfbb44e77101c50f5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de07c72eebb45bfbb44e77101c50f5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43829 } }
I20260619 16:45:31.664992 20141 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7de07c72eebb45bfbb44e77101c50f5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:45:31.665079 20141 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7de07c72eebb45bfbb44e77101c50f5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:45:31.665239 20141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7de07c72eebb45bfbb44e77101c50f5e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:31.666164 20141 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7de07c72eebb45bfbb44e77101c50f5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de07c72eebb45bfbb44e77101c50f5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43829 } }
I20260619 16:45:31.666515 20141 leader_election.cc:302] T 00000000000000000000000000000000 P 7de07c72eebb45bfbb44e77101c50f5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7de07c72eebb45bfbb44e77101c50f5e; no voters:
I20260619 16:45:31.666860 20141 leader_election.cc:288] T 00000000000000000000000000000000 P 7de07c72eebb45bfbb44e77101c50f5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:45:31.667106 20146 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7de07c72eebb45bfbb44e77101c50f5e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:31.667596 20146 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7de07c72eebb45bfbb44e77101c50f5e [term 1 LEADER]: Becoming Leader. State: Replica: 7de07c72eebb45bfbb44e77101c50f5e, State: Running, Role: LEADER
I20260619 16:45:31.667968 20146 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7de07c72eebb45bfbb44e77101c50f5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de07c72eebb45bfbb44e77101c50f5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43829 } }
I20260619 16:45:31.668191 20141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7de07c72eebb45bfbb44e77101c50f5e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:45:31.669552 20148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7de07c72eebb45bfbb44e77101c50f5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7de07c72eebb45bfbb44e77101c50f5e. Latest consensus state: current_term: 1 leader_uuid: "7de07c72eebb45bfbb44e77101c50f5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de07c72eebb45bfbb44e77101c50f5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43829 } } }
I20260619 16:45:31.669687 20148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7de07c72eebb45bfbb44e77101c50f5e [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:31.670063 20156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:45:31.670035 20147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7de07c72eebb45bfbb44e77101c50f5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7de07c72eebb45bfbb44e77101c50f5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de07c72eebb45bfbb44e77101c50f5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43829 } } }
I20260619 16:45:31.670168 20147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7de07c72eebb45bfbb44e77101c50f5e [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:31.673053 20156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:45:31.678960 20156 catalog_manager.cc:1347] Generated new cluster ID: 7e469e0090e14570aa663d98c5874ea0
I20260619 16:45:31.679059 20156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:45:31.721611 20156 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:45:31.722645 20156 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:45:31.736124 20156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7de07c72eebb45bfbb44e77101c50f5e: Generated new TSK 0
I20260619 16:45:31.736941 20156 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:45:31.750859 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43829
--builtin_ntp_servers=127.0.98.20:45087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:45:31.867794 20165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:31.868134 20165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:31.868185 20165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:31.872920 20165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:31.872994 20165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:31.873054 20165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:45:31.878301 20165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:31.879514 20165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:31.881002 20165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:31.891028 20171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:31.891124 20165 server_base.cc:1034] running on GCE node
W20260619 16:45:31.891050 20170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:31.891028 20173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:31.891844 20165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:31.892570 20165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:31.893788 20165 hybrid_clock.cc:648] HybridClock initialized: now 1781887531893756 us; error 47 us; skew 500 ppm
I20260619 16:45:31.895918 20165 webserver.cc:460] Webserver started at http://127.0.98.1:36589/ using document root <none> and password file <none>
I20260619 16:45:31.896530 20165 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:31.896624 20165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:31.896871 20165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:31.899099 20165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-0/data/instance:
uuid: "c6782b9fc8c34e26a10c7b7316ffd504"
format_stamp: "Formatted at 2026-06-19 16:45:31 on dist-test-slave-03cd"
server_key: "f844bd8bf8736c8dc3b4c04b20a604bd"
server_key_iv: "15a7bc66d8f42753e9129a6d92ffd72a"
server_key_version: "encryptionkey@0"
I20260619 16:45:31.899777 20165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c6782b9fc8c34e26a10c7b7316ffd504"
format_stamp: "Formatted at 2026-06-19 16:45:31 on dist-test-slave-03cd"
server_key: "f844bd8bf8736c8dc3b4c04b20a604bd"
server_key_iv: "15a7bc66d8f42753e9129a6d92ffd72a"
server_key_version: "encryptionkey@0"
I20260619 16:45:31.903590 20165 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:45:31.906348 20179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:31.908083 20165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 16:45:31.908263 20165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-0/wal
uuid: "c6782b9fc8c34e26a10c7b7316ffd504"
format_stamp: "Formatted at 2026-06-19 16:45:31 on dist-test-slave-03cd"
server_key: "f844bd8bf8736c8dc3b4c04b20a604bd"
server_key_iv: "15a7bc66d8f42753e9129a6d92ffd72a"
server_key_version: "encryptionkey@0"
I20260619 16:45:31.908420 20165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:31.929769 20165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:31.930635 20165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:31.930886 20165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:31.931602 20165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:45:31.932677 20165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:45:31.932758 20165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:31.932833 20165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:45:31.932890 20165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:31.944172 20165 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43393
I20260619 16:45:31.944188 20292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43393 every 8 connection(s)
I20260619 16:45:31.945232 20165 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:45:31.948148 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 20165
I20260619 16:45:31.948283 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:45:31.948606 392 external_mini_cluster.cc:1428] Setting key d26e97a1d25946a7e99eea610a8c2e97
I20260619 16:45:31.950826 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43829
--builtin_ntp_servers=127.0.98.20:45087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:45:31.956084 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:31.947315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49467 (local address 127.0.98.62:43829)
0619 16:45:31.947493 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:31.947497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:31.948860 (+ 1363us) server_negotiation.cc:407] Connection header received
0619 16:45:31.950437 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:31.950440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:31.950490 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:31.950566 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:31.951912 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.952467 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.953732 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.953973 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.954092 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:31.955481 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:31.955485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:31.955487 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:31.955501 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:31.955521 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:31.955530 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:31.955615 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:31.955844 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 16:45:31.955926 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 16:45:31.957000 20295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:31.947758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43829 (local address 127.0.98.1:49467)
0619 16:45:31.948632 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:31.948699 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:31.950191 (+ 1492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:31.950801 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:31.950812 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:31.950933 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:45:31.951712 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:31.951735 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.952628 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.952632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:31.953569 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:31.953579 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:31.954152 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:31.954157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:31.954362 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:31.955153 (+ 791us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:45:31.955155 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:45:31.955198 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:45:31.955279 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:45:31.955618 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:45:31.955630 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:45:31.955636 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:45:31.955833 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 16:45:31.956148 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":130,"threads_started":1}
I20260619 16:45:31.958415 20293 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43829
I20260619 16:45:31.958675 20293 heartbeater.cc:464] Registering TS with master...
I20260619 16:45:31.959326 20293 heartbeater.cc:511] Master 127.0.98.62:43829 requested a full tablet report, sending...
I20260619 16:45:31.961233 20105 ts_manager.cc:194] Registered new tserver with Master: c6782b9fc8c34e26a10c7b7316ffd504 (127.0.98.1:43393)
I20260619 16:45:31.962872 20105 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:49467
W20260619 16:45:32.071218 20296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:32.071527 20296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:32.071568 20296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:32.075856 20296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:32.075934 20296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:32.075992 20296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:45:32.081248 20296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:32.082386 20296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:32.083813 20296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:32.095010 20304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:32.095010 20301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:32.095162 20302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:32.095501 20296 server_base.cc:1034] running on GCE node
I20260619 16:45:32.096227 20296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:32.096989 20296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:32.098229 20296 hybrid_clock.cc:648] HybridClock initialized: now 1781887532098212 us; error 40 us; skew 500 ppm
I20260619 16:45:32.100435 20296 webserver.cc:460] Webserver started at http://127.0.98.2:33967/ using document root <none> and password file <none>
I20260619 16:45:32.101070 20296 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:32.101168 20296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:32.101426 20296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:32.103742 20296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-1/data/instance:
uuid: "16ecd18f84ff47b3a2aa132b3925c6d1"
format_stamp: "Formatted at 2026-06-19 16:45:32 on dist-test-slave-03cd"
server_key: "5e9654563e40e7085848d599fcdc745a"
server_key_iv: "30a4aae7746740fcea18a5f445fb8a57"
server_key_version: "encryptionkey@0"
I20260619 16:45:32.104355 20296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-1/wal/instance:
uuid: "16ecd18f84ff47b3a2aa132b3925c6d1"
format_stamp: "Formatted at 2026-06-19 16:45:32 on dist-test-slave-03cd"
server_key: "5e9654563e40e7085848d599fcdc745a"
server_key_iv: "30a4aae7746740fcea18a5f445fb8a57"
server_key_version: "encryptionkey@0"
I20260619 16:45:32.108886 20296 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:45:32.111963 20310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:32.113543 20296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:45:32.113700 20296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-1/wal
uuid: "16ecd18f84ff47b3a2aa132b3925c6d1"
format_stamp: "Formatted at 2026-06-19 16:45:32 on dist-test-slave-03cd"
server_key: "5e9654563e40e7085848d599fcdc745a"
server_key_iv: "30a4aae7746740fcea18a5f445fb8a57"
server_key_version: "encryptionkey@0"
I20260619 16:45:32.113848 20296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:32.134480 20296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:32.135311 20296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:32.135533 20296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:32.136240 20296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:45:32.137245 20296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:45:32.137317 20296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:32.137437 20296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:45:32.137490 20296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:32.147616 20296 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40239
I20260619 16:45:32.147640 20423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40239 every 8 connection(s)
I20260619 16:45:32.148703 20296 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:45:32.158567 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 20296
I20260619 16:45:32.158701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:45:32.158991 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:32.150702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34581 (local address 127.0.98.62:43829)
0619 16:45:32.150882 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:32.150887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:32.152062 (+ 1175us) server_negotiation.cc:407] Connection header received
0619 16:45:32.153529 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:32.153532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:32.153585 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:32.153672 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:32.154666 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.155196 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.156686 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.156920 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.157056 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:32.158323 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:32.158326 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:32.158328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:32.158361 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:32.158383 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:32.158392 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:32.158467 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:32.158757 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 16:45:32.158847 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 16:45:32.159014 392 external_mini_cluster.cc:1428] Setting key 74bc7e7c146acd227262ffb3d6f65e70
I20260619 16:45:32.160302 20426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:32.151147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43829 (local address 127.0.98.2:34581)
0619 16:45:32.151910 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:32.151954 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:32.153299 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:32.153880 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:32.153897 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:32.153983 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:45:32.154505 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:32.154518 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.155345 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.155349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:32.156537 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:32.156549 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.157109 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.157113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:32.157289 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:32.158004 (+ 715us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:45:32.158005 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:45:32.158050 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:45:32.158124 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:45:32.158498 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:45:32.158508 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:45:32.158514 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:45:32.158719 (+ 205us) client_negotiation.cc:240] Negotiation successful
0619 16:45:32.159062 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":134,"threads_started":1}
I20260619 16:45:32.161270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43829
--builtin_ntp_servers=127.0.98.20:45087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:45:32.161654 20424 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43829
I20260619 16:45:32.161971 20424 heartbeater.cc:464] Registering TS with master...
I20260619 16:45:32.162626 20424 heartbeater.cc:511] Master 127.0.98.62:43829 requested a full tablet report, sending...
I20260619 16:45:32.163954 20105 ts_manager.cc:194] Registered new tserver with Master: 16ecd18f84ff47b3a2aa132b3925c6d1 (127.0.98.2:40239)
I20260619 16:45:32.164565 20105 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:34581
W20260619 16:45:32.282487 20427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:32.282809 20427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:32.282850 20427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:32.287129 20427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:32.287220 20427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:32.287352 20427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:45:32.292622 20427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:32.293785 20427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:32.295140 20427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:32.305840 20432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:32.305989 20433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:32.306013 20427 server_base.cc:1034] running on GCE node
W20260619 16:45:32.306370 20435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:32.306897 20427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:32.307627 20427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:32.308881 20427 hybrid_clock.cc:648] HybridClock initialized: now 1781887532308859 us; error 38 us; skew 500 ppm
I20260619 16:45:32.310869 20427 webserver.cc:460] Webserver started at http://127.0.98.3:34489/ using document root <none> and password file <none>
I20260619 16:45:32.311486 20427 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:32.311581 20427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:32.311856 20427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:32.314240 20427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-2/data/instance:
uuid: "4838f03f478641b7ae1075862caa73e2"
format_stamp: "Formatted at 2026-06-19 16:45:32 on dist-test-slave-03cd"
server_key: "b706469f2f2465e6d94f1dcf893ab0c1"
server_key_iv: "284432e64256fd5dc885f591a76039d1"
server_key_version: "encryptionkey@0"
I20260619 16:45:32.314873 20427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4838f03f478641b7ae1075862caa73e2"
format_stamp: "Formatted at 2026-06-19 16:45:32 on dist-test-slave-03cd"
server_key: "b706469f2f2465e6d94f1dcf893ab0c1"
server_key_iv: "284432e64256fd5dc885f591a76039d1"
server_key_version: "encryptionkey@0"
I20260619 16:45:32.319815 20427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 16:45:32.322540 20441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:32.324039 20427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:45:32.324203 20427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-2/wal
uuid: "4838f03f478641b7ae1075862caa73e2"
format_stamp: "Formatted at 2026-06-19 16:45:32 on dist-test-slave-03cd"
server_key: "b706469f2f2465e6d94f1dcf893ab0c1"
server_key_iv: "284432e64256fd5dc885f591a76039d1"
server_key_version: "encryptionkey@0"
I20260619 16:45:32.324343 20427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:32.351699 20427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:32.352475 20427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:32.352708 20427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:32.353330 20427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:45:32.354317 20427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:45:32.354391 20427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:32.354461 20427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:45:32.354511 20427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:32.364540 20427 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34303
I20260619 16:45:32.364630 20554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34303 every 8 connection(s)
I20260619 16:45:32.365620 20427 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:45:32.369481 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 20427
I20260619 16:45:32.369624 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:45:32.369962 392 external_mini_cluster.cc:1428] Setting key 9d2c6cb5050e4fccf36537e5a3109aeb
I20260619 16:45:32.376700 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:32.367908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57341 (local address 127.0.98.62:43829)
0619 16:45:32.368066 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:32.368071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:32.369531 (+ 1460us) server_negotiation.cc:407] Connection header received
0619 16:45:32.371236 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:32.371239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:32.371320 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:32.371411 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:32.372691 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.373260 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.374537 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.374781 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.374872 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:32.376161 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:32.376164 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:32.376166 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:32.376181 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:32.376202 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:32.376211 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:32.376280 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:32.376471 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 16:45:32.376533 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:45:32.377586 20557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:32.368351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43829 (local address 127.0.98.3:57341)
0619 16:45:32.369272 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:32.369334 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:32.370969 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:32.371700 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:32.371710 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:32.371833 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:45:32.372525 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:32.372544 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.373408 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.373411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:32.374406 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:32.374416 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.374946 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.374949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:32.375114 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:32.375871 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:45:32.375873 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:45:32.375917 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:45:32.375991 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:45:32.376295 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:45:32.376303 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:45:32.376307 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:45:32.376405 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 16:45:32.376648 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":157,"threads_started":1}
I20260619 16:45:32.378829 20555 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43829
I20260619 16:45:32.379069 20555 heartbeater.cc:464] Registering TS with master...
I20260619 16:45:32.379602 20555 heartbeater.cc:511] Master 127.0.98.62:43829 requested a full tablet report, sending...
I20260619 16:45:32.380792 20105 ts_manager.cc:194] Registered new tserver with Master: 4838f03f478641b7ae1075862caa73e2 (127.0.98.3:34303)
I20260619 16:45:32.381373 20105 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57341
I20260619 16:45:32.384922 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:45:32.391649 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:32.387145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44522 (local address 127.0.98.62:43829)
0619 16:45:32.387305 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:32.387312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:32.387332 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:45:32.387490 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:32.387493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:32.387550 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:32.387643 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:32.388113 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.388655 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.389722 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.389961 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.390057 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:32.391059 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:32.391071 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:32.391073 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:32.391092 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:32.391113 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:32.391121 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:32.391192 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:32.391397 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 16:45:32.391465 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 16:45:32.395239 20105 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:45:32.398543 20105 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:45:32.415529 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:32.409652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58072 (local address 127.0.98.2:40239)
0619 16:45:32.410219 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:32.410227 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:32.410246 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:45:32.410303 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:32.410306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:32.410501 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:32.410646 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:32.410945 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.412222 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.413598 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.414406 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.414522 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:32.414732 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:32.415214 (+ 482us) server_negotiation.cc:299] Negotiation successful
0619 16:45:32.415327 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":134,"threads_started":1}
I20260619 16:45:32.415632 20568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:32.408328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43393 (local address 127.0.0.1:56778)
0619 16:45:32.408962 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:32.409021 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:32.409318 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:32.409945 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:32.409951 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:32.409973 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:32.410117 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:32.410124 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.411598 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.411604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:32.413400 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:32.413412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.414566 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.414573 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:32.414814 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:32.414918 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 16:45:32.415096 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 16:45:32.415341 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":233,"thread_start_us":86,"threads_started":1}
I20260619 16:45:32.416131 20567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:32.408448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56778 (local address 127.0.98.1:43393)
0619 16:45:32.409453 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:32.409459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:32.409481 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:45:32.409547 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:32.409551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:32.409766 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:32.410288 (+ 522us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:32.410451 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.411484 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.413544 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.414406 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.414522 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:32.414732 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:32.415855 (+ 1123us) server_negotiation.cc:299] Negotiation successful
0619 16:45:32.415976 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":893,"thread_start_us":72,"threads_started":1}
I20260619 16:45:32.416157 20570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:32.409108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40239 (local address 127.0.0.1:58072)
0619 16:45:32.409802 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:32.409818 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:32.409901 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:32.410650 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:32.410652 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:32.410671 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:32.410817 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:32.410823 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.412360 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.412363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:32.413436 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:32.413445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.414817 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.414820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:32.414907 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:32.414930 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:45:32.415044 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 16:45:32.416014 (+ 970us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":820,"thread_start_us":118,"threads_started":1}
I20260619 16:45:32.417186 20566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:32.408422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34303 (local address 127.0.0.1:44110)
0619 16:45:32.409244 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:32.409259 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:32.409358 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:32.410073 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:32.410076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:32.410091 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:32.410276 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:32.410283 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.411643 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.411645 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:32.413400 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:32.413412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.416710 (+ 3298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.416714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:32.416810 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:32.416833 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:45:32.416971 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 16:45:32.417062 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":109,"thread_start_us":111,"threads_started":1}
I20260619 16:45:32.419154 20224 tablet_service.cc:1467] Processing CreateTablet for tablet bcd60de0c02e4418ac9e4b3d9b853f87 (DEFAULT_TABLE table=test-table [id=d6e57d00e5e24cde997eeb431b848f17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:45:32.419153 20358 tablet_service.cc:1467] Processing CreateTablet for tablet bcd60de0c02e4418ac9e4b3d9b853f87 (DEFAULT_TABLE table=test-table [id=d6e57d00e5e24cde997eeb431b848f17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:45:32.420259 20358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcd60de0c02e4418ac9e4b3d9b853f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:32.420259 20224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcd60de0c02e4418ac9e4b3d9b853f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:32.424299 20569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:32.408811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44110 (local address 127.0.98.3:34303)
0619 16:45:32.409244 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:32.409259 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:32.409456 (+ 197us) server_negotiation.cc:407] Connection header received
0619 16:45:32.409542 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:32.409547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:32.409766 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:32.409923 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:32.410406 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.411536 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.414806 (+ 3270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.415625 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.416417 (+ 792us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:32.416631 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:32.423904 (+ 7273us) server_negotiation.cc:299] Negotiation successful
0619 16:45:32.424078 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":134,"threads_started":1}
I20260619 16:45:32.427560 20572 tablet_bootstrap.cc:492] T bcd60de0c02e4418ac9e4b3d9b853f87 P c6782b9fc8c34e26a10c7b7316ffd504: Bootstrap starting.
I20260619 16:45:32.427577 20489 tablet_service.cc:1467] Processing CreateTablet for tablet bcd60de0c02e4418ac9e4b3d9b853f87 (DEFAULT_TABLE table=test-table [id=d6e57d00e5e24cde997eeb431b848f17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:45:32.428200 20573 tablet_bootstrap.cc:492] T bcd60de0c02e4418ac9e4b3d9b853f87 P 16ecd18f84ff47b3a2aa132b3925c6d1: Bootstrap starting.
I20260619 16:45:32.429299 20489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcd60de0c02e4418ac9e4b3d9b853f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:32.430191 20573 tablet_bootstrap.cc:654] T bcd60de0c02e4418ac9e4b3d9b853f87 P 16ecd18f84ff47b3a2aa132b3925c6d1: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:32.430258 20572 tablet_bootstrap.cc:654] T bcd60de0c02e4418ac9e4b3d9b853f87 P c6782b9fc8c34e26a10c7b7316ffd504: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:32.430975 20573 log.cc:824] T bcd60de0c02e4418ac9e4b3d9b853f87 P 16ecd18f84ff47b3a2aa132b3925c6d1: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:32.431070 20572 log.cc:824] T bcd60de0c02e4418ac9e4b3d9b853f87 P c6782b9fc8c34e26a10c7b7316ffd504: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:32.433166 20573 tablet_bootstrap.cc:492] T bcd60de0c02e4418ac9e4b3d9b853f87 P 16ecd18f84ff47b3a2aa132b3925c6d1: No bootstrap required, opened a new log
I20260619 16:45:32.433166 20572 tablet_bootstrap.cc:492] T bcd60de0c02e4418ac9e4b3d9b853f87 P c6782b9fc8c34e26a10c7b7316ffd504: No bootstrap required, opened a new log
I20260619 16:45:32.433382 20572 ts_tablet_manager.cc:1397] T bcd60de0c02e4418ac9e4b3d9b853f87 P c6782b9fc8c34e26a10c7b7316ffd504: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 16:45:32.433386 20573 ts_tablet_manager.cc:1397] T bcd60de0c02e4418ac9e4b3d9b853f87 P 16ecd18f84ff47b3a2aa132b3925c6d1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 16:45:32.436918 20573 raft_consensus.cc:348] T bcd60de0c02e4418ac9e4b3d9b853f87 P 16ecd18f84ff47b3a2aa132b3925c6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6782b9fc8c34e26a10c7b7316ffd504" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "4838f03f478641b7ae1075862caa73e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34303 } } peers { permanent_uuid: "16ecd18f84ff47b3a2aa132b3925c6d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40239 } }
I20260619 16:45:32.437201 20573 raft_consensus.cc:374] T bcd60de0c02e4418ac9e4b3d9b853f87 P 16ecd18f84ff47b3a2aa132b3925c6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:32.437253 20573 raft_consensus.cc:729] T bcd60de0c02e4418ac9e4b3d9b853f87 P 16ecd18f84ff47b3a2aa132b3925c6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16ecd18f84ff47b3a2aa132b3925c6d1, State: Initialized, Role: FOLLOWER
I20260619 16:45:32.437731 20576 tablet_bootstrap.cc:492] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2: Bootstrap starting.
I20260619 16:45:32.437803 20573 consensus_queue.cc:260] T bcd60de0c02e4418ac9e4b3d9b853f87 P 16ecd18f84ff47b3a2aa132b3925c6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6782b9fc8c34e26a10c7b7316ffd504" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "4838f03f478641b7ae1075862caa73e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34303 } } peers { permanent_uuid: "16ecd18f84ff47b3a2aa132b3925c6d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40239 } }
I20260619 16:45:32.437737 20572 raft_consensus.cc:348] T bcd60de0c02e4418ac9e4b3d9b853f87 P c6782b9fc8c34e26a10c7b7316ffd504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6782b9fc8c34e26a10c7b7316ffd504" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "4838f03f478641b7ae1075862caa73e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34303 } } peers { permanent_uuid: "16ecd18f84ff47b3a2aa132b3925c6d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40239 } }
I20260619 16:45:32.438005 20572 raft_consensus.cc:374] T bcd60de0c02e4418ac9e4b3d9b853f87 P c6782b9fc8c34e26a10c7b7316ffd504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:32.438061 20572 raft_consensus.cc:729] T bcd60de0c02e4418ac9e4b3d9b853f87 P c6782b9fc8c34e26a10c7b7316ffd504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6782b9fc8c34e26a10c7b7316ffd504, State: Initialized, Role: FOLLOWER
I20260619 16:45:32.438556 20572 consensus_queue.cc:260] T bcd60de0c02e4418ac9e4b3d9b853f87 P c6782b9fc8c34e26a10c7b7316ffd504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6782b9fc8c34e26a10c7b7316ffd504" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "4838f03f478641b7ae1075862caa73e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34303 } } peers { permanent_uuid: "16ecd18f84ff47b3a2aa132b3925c6d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40239 } }
I20260619 16:45:32.438758 20424 heartbeater.cc:503] Master 127.0.98.62:43829 was elected leader, sending a full tablet report...
I20260619 16:45:32.439199 20293 heartbeater.cc:503] Master 127.0.98.62:43829 was elected leader, sending a full tablet report...
I20260619 16:45:32.440454 20572 ts_tablet_manager.cc:1428] T bcd60de0c02e4418ac9e4b3d9b853f87 P c6782b9fc8c34e26a10c7b7316ffd504: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 16:45:32.441197 20576 tablet_bootstrap.cc:654] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:32.442227 20576 log.cc:824] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:32.443066 20573 ts_tablet_manager.cc:1428] T bcd60de0c02e4418ac9e4b3d9b853f87 P 16ecd18f84ff47b3a2aa132b3925c6d1: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260619 16:45:32.444672 20576 tablet_bootstrap.cc:492] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2: No bootstrap required, opened a new log
I20260619 16:45:32.444862 20576 ts_tablet_manager.cc:1397] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
W20260619 16:45:32.446528 20294 tablet.cc:2369] T bcd60de0c02e4418ac9e4b3d9b853f87 P c6782b9fc8c34e26a10c7b7316ffd504: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:45:32.448580 20576 raft_consensus.cc:348] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6782b9fc8c34e26a10c7b7316ffd504" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "4838f03f478641b7ae1075862caa73e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34303 } } peers { permanent_uuid: "16ecd18f84ff47b3a2aa132b3925c6d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40239 } }
I20260619 16:45:32.448825 20576 raft_consensus.cc:374] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:32.448866 20576 raft_consensus.cc:729] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4838f03f478641b7ae1075862caa73e2, State: Initialized, Role: FOLLOWER
I20260619 16:45:32.449239 20576 consensus_queue.cc:260] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6782b9fc8c34e26a10c7b7316ffd504" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "4838f03f478641b7ae1075862caa73e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34303 } } peers { permanent_uuid: "16ecd18f84ff47b3a2aa132b3925c6d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40239 } }
I20260619 16:45:32.449967 20555 heartbeater.cc:503] Master 127.0.98.62:43829 was elected leader, sending a full tablet report...
I20260619 16:45:32.450278 20576 ts_tablet_manager.cc:1428] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260619 16:45:32.616756 20556 tablet.cc:2369] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:45:32.635296 20580 raft_consensus.cc:482] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:45:32.635548 20580 raft_consensus.cc:504] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6782b9fc8c34e26a10c7b7316ffd504" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "4838f03f478641b7ae1075862caa73e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34303 } } peers { permanent_uuid: "16ecd18f84ff47b3a2aa132b3925c6d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40239 } }
I20260619 16:45:32.636795 20580 leader_election.cc:288] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6782b9fc8c34e26a10c7b7316ffd504 (127.0.98.1:43393), 16ecd18f84ff47b3a2aa132b3925c6d1 (127.0.98.2:40239)
I20260619 16:45:32.640756 20557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:32.637011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43393 (local address 127.0.98.3:55417)
0619 16:45:32.637214 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:32.637230 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:32.637351 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:32.637686 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:32.637689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:32.637708 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:32.637873 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:32.637882 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.638674 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.638677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:32.639592 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:32.639602 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.640301 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.640304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:32.640407 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:32.640435 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 16:45:32.640532 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 16:45:32.640612 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":33}
I20260619 16:45:32.640756 20567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:32.637106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55417 (local address 127.0.98.1:43393)
0619 16:45:32.637250 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:32.637255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:32.637278 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:45:32.637488 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:32.637494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:32.637558 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:32.637641 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:32.637998 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.638559 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.639769 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.640168 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.640294 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:32.640372 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:32.640549 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 16:45:32.640623 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:45:32.640811 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:32.637151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48977 (local address 127.0.98.2:40239)
0619 16:45:32.637277 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:32.637281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:32.637579 (+ 298us) server_negotiation.cc:407] Connection header received
0619 16:45:32.637709 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:32.637712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:32.637762 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:32.637845 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:32.638142 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.638670 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.639828 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.640213 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.640335 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:32.640394 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:32.640582 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 16:45:32.640654 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:45:32.640794 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:32.637038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40239 (local address 127.0.98.3:48977)
0619 16:45:32.637473 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:32.637488 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:32.637601 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:32.637861 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:32.637863 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:32.637877 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:32.638021 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:32.638028 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.638777 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.638779 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:32.639690 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:32.639703 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.640338 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.640342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:32.640441 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:32.640471 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 16:45:32.640564 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 16:45:32.640634 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":17,"thread_start_us":106,"threads_started":1}
I20260619 16:45:32.641637 20247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd60de0c02e4418ac9e4b3d9b853f87" candidate_uuid: "4838f03f478641b7ae1075862caa73e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6782b9fc8c34e26a10c7b7316ffd504" is_pre_election: true
I20260619 16:45:32.641862 20378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd60de0c02e4418ac9e4b3d9b853f87" candidate_uuid: "4838f03f478641b7ae1075862caa73e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16ecd18f84ff47b3a2aa132b3925c6d1" is_pre_election: true
I20260619 16:45:32.641984 20247 raft_consensus.cc:2454] T bcd60de0c02e4418ac9e4b3d9b853f87 P c6782b9fc8c34e26a10c7b7316ffd504 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4838f03f478641b7ae1075862caa73e2 in term 0.
I20260619 16:45:32.642186 20378 raft_consensus.cc:2454] T bcd60de0c02e4418ac9e4b3d9b853f87 P 16ecd18f84ff47b3a2aa132b3925c6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4838f03f478641b7ae1075862caa73e2 in term 0.
I20260619 16:45:32.642556 20444 leader_election.cc:302] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4838f03f478641b7ae1075862caa73e2, c6782b9fc8c34e26a10c7b7316ffd504; no voters:
I20260619 16:45:32.642895 20580 raft_consensus.cc:2781] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:45:32.642975 20580 raft_consensus.cc:482] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:45:32.643025 20580 raft_consensus.cc:3037] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:32.644161 20580 raft_consensus.cc:504] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6782b9fc8c34e26a10c7b7316ffd504" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "4838f03f478641b7ae1075862caa73e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34303 } } peers { permanent_uuid: "16ecd18f84ff47b3a2aa132b3925c6d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40239 } }
I20260619 16:45:32.644588 20580 leader_election.cc:288] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2 [CANDIDATE]: Term 1 election: Requested vote from peers c6782b9fc8c34e26a10c7b7316ffd504 (127.0.98.1:43393), 16ecd18f84ff47b3a2aa132b3925c6d1 (127.0.98.2:40239)
I20260619 16:45:32.644971 20378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd60de0c02e4418ac9e4b3d9b853f87" candidate_uuid: "4838f03f478641b7ae1075862caa73e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16ecd18f84ff47b3a2aa132b3925c6d1"
I20260619 16:45:32.644975 20247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd60de0c02e4418ac9e4b3d9b853f87" candidate_uuid: "4838f03f478641b7ae1075862caa73e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6782b9fc8c34e26a10c7b7316ffd504"
I20260619 16:45:32.645113 20247 raft_consensus.cc:3037] T bcd60de0c02e4418ac9e4b3d9b853f87 P c6782b9fc8c34e26a10c7b7316ffd504 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:32.645113 20378 raft_consensus.cc:3037] T bcd60de0c02e4418ac9e4b3d9b853f87 P 16ecd18f84ff47b3a2aa132b3925c6d1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:32.646193 20378 raft_consensus.cc:2454] T bcd60de0c02e4418ac9e4b3d9b853f87 P 16ecd18f84ff47b3a2aa132b3925c6d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4838f03f478641b7ae1075862caa73e2 in term 1.
I20260619 16:45:32.646417 20247 raft_consensus.cc:2454] T bcd60de0c02e4418ac9e4b3d9b853f87 P c6782b9fc8c34e26a10c7b7316ffd504 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4838f03f478641b7ae1075862caa73e2 in term 1.
I20260619 16:45:32.646572 20445 leader_election.cc:302] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16ecd18f84ff47b3a2aa132b3925c6d1, 4838f03f478641b7ae1075862caa73e2; no voters:
I20260619 16:45:32.646760 20580 raft_consensus.cc:2781] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:32.646950 20580 raft_consensus.cc:686] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2 [term 1 LEADER]: Becoming Leader. State: Replica: 4838f03f478641b7ae1075862caa73e2, State: Running, Role: LEADER
I20260619 16:45:32.647269 20580 consensus_queue.cc:237] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6782b9fc8c34e26a10c7b7316ffd504" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "4838f03f478641b7ae1075862caa73e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34303 } } peers { permanent_uuid: "16ecd18f84ff47b3a2aa132b3925c6d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40239 } }
W20260619 16:45:32.650117 20425 tablet.cc:2369] T bcd60de0c02e4418ac9e4b3d9b853f87 P 16ecd18f84ff47b3a2aa132b3925c6d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:45:32.650782 20105 catalog_manager.cc:5495] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4838f03f478641b7ae1075862caa73e2 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4838f03f478641b7ae1075862caa73e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6782b9fc8c34e26a10c7b7316ffd504" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4838f03f478641b7ae1075862caa73e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16ecd18f84ff47b3a2aa132b3925c6d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40239 } health_report { overall_health: UNKNOWN } } }
I20260619 16:45:32.718220 20569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:32.714088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44118 (local address 127.0.98.3:34303)
0619 16:45:32.714234 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:32.714237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:32.714252 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:45:32.714354 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:32.714356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:32.714409 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:32.714502 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:45:32.714720 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.715215 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.716251 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:32.716485 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:32.716565 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:32.716954 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:45:32.717497 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:45:32.717652 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:32.717890 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 16:45:32.718006 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:45:32.724923 20247 raft_consensus.cc:1261] T bcd60de0c02e4418ac9e4b3d9b853f87 P c6782b9fc8c34e26a10c7b7316ffd504 [term 1 FOLLOWER]: Refusing update from remote peer 4838f03f478641b7ae1075862caa73e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:45:32.724923 20378 raft_consensus.cc:1261] T bcd60de0c02e4418ac9e4b3d9b853f87 P 16ecd18f84ff47b3a2aa132b3925c6d1 [term 1 FOLLOWER]: Refusing update from remote peer 4838f03f478641b7ae1075862caa73e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:45:32.725780 20582 consensus_queue.cc:1035] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6782b9fc8c34e26a10c7b7316ffd504" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:45:32.726012 20580 consensus_queue.cc:1035] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16ecd18f84ff47b3a2aa132b3925c6d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:45:32.734295 20586 mvcc.cc:205] Tried to move back new op lower bound from 7298611334032371712 to 7298611333726126080. Current Snapshot: MvccSnapshot[applied={T|T < 7298611334032371712}]
I20260619 16:45:32.742899 20105 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:45:32.743073 20105 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:45:32.745749 20105 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7de07c72eebb45bfbb44e77101c50f5e: Sending DeleteTablet for 3 replicas of tablet bcd60de0c02e4418ac9e4b3d9b853f87
I20260619 16:45:32.746623 20224 tablet_service.cc:1514] Processing DeleteTablet for tablet bcd60de0c02e4418ac9e4b3d9b853f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:45:32 UTC) from {username='slave'} at 127.0.0.1:56778
I20260619 16:45:32.746735 20489 tablet_service.cc:1514] Processing DeleteTablet for tablet bcd60de0c02e4418ac9e4b3d9b853f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:45:32 UTC) from {username='slave'} at 127.0.0.1:44110
I20260619 16:45:32.746804 20358 tablet_service.cc:1514] Processing DeleteTablet for tablet bcd60de0c02e4418ac9e4b3d9b853f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:45:32 UTC) from {username='slave'} at 127.0.0.1:58072
I20260619 16:45:32.747265 20596 tablet_replica.cc:330] T bcd60de0c02e4418ac9e4b3d9b853f87 P 16ecd18f84ff47b3a2aa132b3925c6d1: stopping tablet replica
I20260619 16:45:32.747370 20595 tablet_replica.cc:330] T bcd60de0c02e4418ac9e4b3d9b853f87 P c6782b9fc8c34e26a10c7b7316ffd504: stopping tablet replica
I20260619 16:45:32.747617 20595 raft_consensus.cc:2229] T bcd60de0c02e4418ac9e4b3d9b853f87 P c6782b9fc8c34e26a10c7b7316ffd504 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:45:32.747617 20596 raft_consensus.cc:2229] T bcd60de0c02e4418ac9e4b3d9b853f87 P 16ecd18f84ff47b3a2aa132b3925c6d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:45:32.748008 20596 raft_consensus.cc:2258] T bcd60de0c02e4418ac9e4b3d9b853f87 P 16ecd18f84ff47b3a2aa132b3925c6d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:45:32.748008 20595 raft_consensus.cc:2258] T bcd60de0c02e4418ac9e4b3d9b853f87 P c6782b9fc8c34e26a10c7b7316ffd504 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:45:32.748425 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 20165
I20260619 16:45:32.750628 20596 ts_tablet_manager.cc:1905] T bcd60de0c02e4418ac9e4b3d9b853f87 P 16ecd18f84ff47b3a2aa132b3925c6d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:45:32.751468 20597 tablet_replica.cc:330] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2: stopping tablet replica
I20260619 16:45:32.751880 20597 raft_consensus.cc:2229] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:45:32.752334 20597 raft_consensus.cc:2258] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:45:32.753952 20597 ts_tablet_manager.cc:1905] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:45:32.754770 20596 ts_tablet_manager.cc:1918] T bcd60de0c02e4418ac9e4b3d9b853f87 P 16ecd18f84ff47b3a2aa132b3925c6d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:45:32.754886 20596 log.cc:1196] T bcd60de0c02e4418ac9e4b3d9b853f87 P 16ecd18f84ff47b3a2aa132b3925c6d1: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-1/wal/wals/bcd60de0c02e4418ac9e4b3d9b853f87
I20260619 16:45:32.755215 20596 ts_tablet_manager.cc:1939] T bcd60de0c02e4418ac9e4b3d9b853f87 P 16ecd18f84ff47b3a2aa132b3925c6d1: Deleting consensus metadata
W20260619 16:45:32.756564 20091 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43393 (error 108)
I20260619 16:45:32.756825 20092 catalog_manager.cc:4841] TS 16ecd18f84ff47b3a2aa132b3925c6d1 (127.0.98.2:40239): tablet bcd60de0c02e4418ac9e4b3d9b853f87 (table test-table [id=d6e57d00e5e24cde997eeb431b848f17]) successfully deleted
I20260619 16:45:32.757444 20597 ts_tablet_manager.cc:1918] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:45:32.757566 20597 log.cc:1196] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2: Deleting WAL directory at /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887259574980-392-0/minicluster-data/ts-2/wal/wals/bcd60de0c02e4418ac9e4b3d9b853f87
I20260619 16:45:32.757588 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 20296
I20260619 16:45:32.758932 20570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:32.758504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43393 (local address 127.0.0.1:56784)
0619 16:45:32.758701 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:32.758825 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 16:45:32.759313 20597 ts_tablet_manager.cc:1939] T bcd60de0c02e4418ac9e4b3d9b853f87 P 4838f03f478641b7ae1075862caa73e2: Deleting consensus metadata
W20260619 16:45:32.759517 20091 catalog_manager.cc:4584] TS c6782b9fc8c34e26a10c7b7316ffd504 (127.0.98.1:43393): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bcd60de0c02e4418ac9e4b3d9b853f87: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43393: connect: Connection refused (error 111)
I20260619 16:45:32.760578 20090 catalog_manager.cc:4841] TS 4838f03f478641b7ae1075862caa73e2 (127.0.98.3:34303): tablet bcd60de0c02e4418ac9e4b3d9b853f87 (table test-table [id=d6e57d00e5e24cde997eeb431b848f17]) successfully deleted
I20260619 16:45:32.763496 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 20427
I20260619 16:45:32.768852 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 20074
2026-06-19T16:45:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1348 ms)
[----------] 6 tests from EncryptionPolicyTest (11759 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:45:32 dist-test-slave-03cd krb5kdc[20601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:45:32 dist-test-slave-03cd krb5kdc[20601](info): set up 2 sockets
Jun 19 16:45:32 dist-test-slave-03cd krb5kdc[20601](info): commencing operation
krb5kdc: starting...
W20260619 16:45:33.852010 392 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:45:33 dist-test-slave-03cd krb5kdc[20601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:45:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:45:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:45:33.984192 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43687
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:34.099923 20614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:34.100327 20614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:34.100419 20614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:34.104864 20614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:45:34.104983 20614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:34.105022 20614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:34.105063 20614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:45:34.105104 20614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:45:34.110929 20614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43687
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:34.112272 20614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:34.113379 20614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:34.123858 20619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:34.123881 20620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:34.123868 20622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:34.124045 20614 server_base.cc:1034] running on GCE node
I20260619 16:45:34.124992 20614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:34.126303 20614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:34.127534 20614 hybrid_clock.cc:648] HybridClock initialized: now 1781887534127509 us; error 45 us; skew 500 ppm
Jun 19 16:45:34 dist-test-slave-03cd krb5kdc[20601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:34.131004 20614 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:45:34.132910 20614 webserver.cc:460] Webserver started at http://127.0.0.1:41499/ using document root <none> and password file <none>
I20260619 16:45:34.133601 20614 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:34.133713 20614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:34.133975 20614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:34.136235 20614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "ada6dd722c024caba895db4b0f82de2c"
format_stamp: "Formatted at 2026-06-19 16:45:34 on dist-test-slave-03cd"
server_key: "8a4367227f9299763dd8bad6791e8202"
server_key_iv: "11751377d5f4dadc35cd5daaf8400d71"
server_key_version: "encryptionkey@0"
I20260619 16:45:34.136852 20614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "ada6dd722c024caba895db4b0f82de2c"
format_stamp: "Formatted at 2026-06-19 16:45:34 on dist-test-slave-03cd"
server_key: "8a4367227f9299763dd8bad6791e8202"
server_key_iv: "11751377d5f4dadc35cd5daaf8400d71"
server_key_version: "encryptionkey@0"
I20260619 16:45:34.140929 20614 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260619 16:45:34.143577 20629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:34.145033 20614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:45:34.145190 20614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "ada6dd722c024caba895db4b0f82de2c"
format_stamp: "Formatted at 2026-06-19 16:45:34 on dist-test-slave-03cd"
server_key: "8a4367227f9299763dd8bad6791e8202"
server_key_iv: "11751377d5f4dadc35cd5daaf8400d71"
server_key_version: "encryptionkey@0"
I20260619 16:45:34.145334 20614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:34.162688 20614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:34.166283 20614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:34.166503 20614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:34.174333 20614 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43687
I20260619 16:45:34.174360 20681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43687 every 8 connection(s)
I20260619 16:45:34.175364 20614 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:45:34.178313 20682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:34.181949 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 20614
I20260619 16:45:34.182106 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:45:34.182420 392 external_mini_cluster.cc:1428] Setting key a0694d0855b8b35c17f290fc5334a828
I20260619 16:45:34.184934 20682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ada6dd722c024caba895db4b0f82de2c: Bootstrap starting.
I20260619 16:45:34.187299 20682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ada6dd722c024caba895db4b0f82de2c: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:34.188169 20682 log.cc:824] T 00000000000000000000000000000000 P ada6dd722c024caba895db4b0f82de2c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:34.190464 20682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ada6dd722c024caba895db4b0f82de2c: No bootstrap required, opened a new log
Jun 19 16:45:34 dist-test-slave-03cd krb5kdc[20601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 16:45:34.194947 20682 raft_consensus.cc:348] T 00000000000000000000000000000000 P ada6dd722c024caba895db4b0f82de2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada6dd722c024caba895db4b0f82de2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } }
I20260619 16:45:34.195243 20682 raft_consensus.cc:374] T 00000000000000000000000000000000 P ada6dd722c024caba895db4b0f82de2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:34.195334 20682 raft_consensus.cc:729] T 00000000000000000000000000000000 P ada6dd722c024caba895db4b0f82de2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada6dd722c024caba895db4b0f82de2c, State: Initialized, Role: FOLLOWER
I20260619 16:45:34.195874 20682 consensus_queue.cc:260] T 00000000000000000000000000000000 P ada6dd722c024caba895db4b0f82de2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada6dd722c024caba895db4b0f82de2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } }
I20260619 16:45:34.196044 20682 raft_consensus.cc:388] T 00000000000000000000000000000000 P ada6dd722c024caba895db4b0f82de2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:45:34.196125 20682 raft_consensus.cc:482] T 00000000000000000000000000000000 P ada6dd722c024caba895db4b0f82de2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:45:34.196257 20682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ada6dd722c024caba895db4b0f82de2c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:34.197321 20682 raft_consensus.cc:504] T 00000000000000000000000000000000 P ada6dd722c024caba895db4b0f82de2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada6dd722c024caba895db4b0f82de2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } }
I20260619 16:45:34.197736 20682 leader_election.cc:302] T 00000000000000000000000000000000 P ada6dd722c024caba895db4b0f82de2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ada6dd722c024caba895db4b0f82de2c; no voters:
I20260619 16:45:34.198107 20682 leader_election.cc:288] T 00000000000000000000000000000000 P ada6dd722c024caba895db4b0f82de2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:45:34.198215 20687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ada6dd722c024caba895db4b0f82de2c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:34.198395 20687 raft_consensus.cc:686] T 00000000000000000000000000000000 P ada6dd722c024caba895db4b0f82de2c [term 1 LEADER]: Becoming Leader. State: Replica: ada6dd722c024caba895db4b0f82de2c, State: Running, Role: LEADER
I20260619 16:45:34.198598 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:34.184014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56717 (local address 127.0.0.1:43687)
0619 16:45:34.184505 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:34.184521 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:34.184558 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:45:34.185317 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:34.185337 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:34.185717 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:34.186082 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:34.187005 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:34.188165 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:34.189386 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:34.189896 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:34.190079 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:34.192992 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:34.192998 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:34.193024 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:34.193057 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:34.194942 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:34.195593 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:34.195600 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:34.195609 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:34.195777 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:34.196199 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:34.196203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:34.196206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:34.196712 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:34.196961 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:34.197270 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 16:45:34.197557 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":155,"threads_started":1}
I20260619 16:45:34.198705 20687 consensus_queue.cc:237] T 00000000000000000000000000000000 P ada6dd722c024caba895db4b0f82de2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada6dd722c024caba895db4b0f82de2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } }
I20260619 16:45:34.199451 20682 sys_catalog.cc:564] T 00000000000000000000000000000000 P ada6dd722c024caba895db4b0f82de2c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:45:34.199927 20687 sys_catalog.cc:455] T 00000000000000000000000000000000 P ada6dd722c024caba895db4b0f82de2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ada6dd722c024caba895db4b0f82de2c. Latest consensus state: current_term: 1 leader_uuid: "ada6dd722c024caba895db4b0f82de2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada6dd722c024caba895db4b0f82de2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } } }
I20260619 16:45:34.200049 20687 sys_catalog.cc:458] T 00000000000000000000000000000000 P ada6dd722c024caba895db4b0f82de2c [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:34.200717 20688 sys_catalog.cc:455] T 00000000000000000000000000000000 P ada6dd722c024caba895db4b0f82de2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ada6dd722c024caba895db4b0f82de2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada6dd722c024caba895db4b0f82de2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } } }
I20260619 16:45:34.200822 20688 sys_catalog.cc:458] T 00000000000000000000000000000000 P ada6dd722c024caba895db4b0f82de2c [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:34.201089 20691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:45:34.202078 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:45:34.204586 20691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:45:34.211532 20691 catalog_manager.cc:1347] Generated new cluster ID: e73b62efcdd848fe995b61a866ffbff4
I20260619 16:45:34.211624 20691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:45:34.218317 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:34.208403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41345 (local address 127.0.0.1:43687)
0619 16:45:34.208517 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:34.208524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:34.209122 (+ 598us) server_negotiation.cc:407] Connection header received
0619 16:45:34.209313 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:34.209316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:34.209374 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:34.209462 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:34.210258 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:34.210808 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:34.212000 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:34.212374 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:34.212497 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:34.214321 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:34.214325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:34.214328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:34.214360 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:34.216292 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:34.216863 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:34.216866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:34.216868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:34.216963 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:34.217335 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:34.217339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:34.217340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:34.217562 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:34.217710 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:34.218031 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 16:45:34.218133 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":33}
I20260619 16:45:34.219591 20691 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:45:34.220551 20691 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:45:34.226432 20691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ada6dd722c024caba895db4b0f82de2c: Generated new TSK 0
I20260619 16:45:34.227373 20691 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:45:34.233000 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 20614
W20260619 16:45:34.237759 20599 connection.cc:535] client connection to 127.0.0.1:43687 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43687): Connection reset by peer (error 104)
2026-06-19T16:45:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1461 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:45:34 dist-test-slave-03cd krb5kdc[20713](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:45:34 dist-test-slave-03cd krb5kdc[20713](info): set up 2 sockets
Jun 19 16:45:34 dist-test-slave-03cd krb5kdc[20713](info): commencing operation
krb5kdc: starting...
W20260619 16:45:35.300401 392 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:45:35 dist-test-slave-03cd krb5kdc[20713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:45:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:45:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:45:35.436614 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41833
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:35.548977 20726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:35.549340 20726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:35.549425 20726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:35.553740 20726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:45:35.553864 20726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:35.553907 20726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:35.553946 20726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:45:35.553978 20726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:45:35.559686 20726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41833
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:35.560904 20726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:35.561969 20726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:35.571646 20734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:35.571625 20731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:35.571641 20732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:35.571990 20726 server_base.cc:1034] running on GCE node
I20260619 16:45:35.572719 20726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:35.573978 20726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:35.575217 20726 hybrid_clock.cc:648] HybridClock initialized: now 1781887535575191 us; error 39 us; skew 500 ppm
Jun 19 16:45:35 dist-test-slave-03cd krb5kdc[20713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:35.578404 20726 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:45:35.579475 20726 webserver.cc:460] Webserver started at http://127.0.0.1:33409/ using document root <none> and password file <none>
I20260619 16:45:35.580114 20726 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:35.580180 20726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:35.580365 20726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:35.582494 20726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "064166c511684e5498919039d85fab8c"
format_stamp: "Formatted at 2026-06-19 16:45:35 on dist-test-slave-03cd"
server_key: "a9ac8c57d5047aea4413c2df0f69f530"
server_key_iv: "ba6fbef81227950427264e2e5310d226"
server_key_version: "encryptionkey@0"
I20260619 16:45:35.583101 20726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "064166c511684e5498919039d85fab8c"
format_stamp: "Formatted at 2026-06-19 16:45:35 on dist-test-slave-03cd"
server_key: "a9ac8c57d5047aea4413c2df0f69f530"
server_key_iv: "ba6fbef81227950427264e2e5310d226"
server_key_version: "encryptionkey@0"
I20260619 16:45:35.587175 20726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:45:35.590307 20741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:35.591799 20726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:45:35.591934 20726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "064166c511684e5498919039d85fab8c"
format_stamp: "Formatted at 2026-06-19 16:45:35 on dist-test-slave-03cd"
server_key: "a9ac8c57d5047aea4413c2df0f69f530"
server_key_iv: "ba6fbef81227950427264e2e5310d226"
server_key_version: "encryptionkey@0"
I20260619 16:45:35.592047 20726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:35.606916 20726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:35.610486 20726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:35.610658 20726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:35.618291 20726 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41833
I20260619 16:45:35.618304 20793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41833 every 8 connection(s)
I20260619 16:45:35.619309 20726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:45:35.622231 20794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:35.623811 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 20726
I20260619 16:45:35.623963 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:45:35.624300 392 external_mini_cluster.cc:1428] Setting key 8386a67dff2e50c06e39e8f52543df1a
I20260619 16:45:35.628767 20794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 064166c511684e5498919039d85fab8c: Bootstrap starting.
I20260619 16:45:35.631044 20794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 064166c511684e5498919039d85fab8c: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:35.631863 20794 log.cc:824] T 00000000000000000000000000000000 P 064166c511684e5498919039d85fab8c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:35.633992 20794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 064166c511684e5498919039d85fab8c: No bootstrap required, opened a new log
I20260619 16:45:35.634658 20797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:35.625812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36855 (local address 127.0.0.1:41833)
0619 16:45:35.626296 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:35.626313 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:35.626352 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:45:35.627112 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:35.627131 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:35.627498 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:35.627987 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:35.628446 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:35.629578 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:35.630696 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:35.631292 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:35.631494 (+ 202us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:35.632586 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:35.632597 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:35.632619 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:35.633071 (+ 452us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:35.633136 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:35.633162 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:35.633275 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:35.633594 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 16:45:35.633747 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":138,"threads_started":1}
I20260619 16:45:35.637774 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:45:35.637755 20794 raft_consensus.cc:348] T 00000000000000000000000000000000 P 064166c511684e5498919039d85fab8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "064166c511684e5498919039d85fab8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41833 } }
I20260619 16:45:35.638033 20794 raft_consensus.cc:374] T 00000000000000000000000000000000 P 064166c511684e5498919039d85fab8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:35.638082 20794 raft_consensus.cc:729] T 00000000000000000000000000000000 P 064166c511684e5498919039d85fab8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 064166c511684e5498919039d85fab8c, State: Initialized, Role: FOLLOWER
I20260619 16:45:35.638546 20794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 064166c511684e5498919039d85fab8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "064166c511684e5498919039d85fab8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41833 } }
I20260619 16:45:35.638741 20794 raft_consensus.cc:388] T 00000000000000000000000000000000 P 064166c511684e5498919039d85fab8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:45:35.638803 20794 raft_consensus.cc:482] T 00000000000000000000000000000000 P 064166c511684e5498919039d85fab8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:45:35.638900 20794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 064166c511684e5498919039d85fab8c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:35.640249 20794 raft_consensus.cc:504] T 00000000000000000000000000000000 P 064166c511684e5498919039d85fab8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "064166c511684e5498919039d85fab8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41833 } }
I20260619 16:45:35.640678 20794 leader_election.cc:302] T 00000000000000000000000000000000 P 064166c511684e5498919039d85fab8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 064166c511684e5498919039d85fab8c; no voters:
I20260619 16:45:35.641083 20794 leader_election.cc:288] T 00000000000000000000000000000000 P 064166c511684e5498919039d85fab8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:45:35.641253 20804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 064166c511684e5498919039d85fab8c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:35.641471 20804 raft_consensus.cc:686] T 00000000000000000000000000000000 P 064166c511684e5498919039d85fab8c [term 1 LEADER]: Becoming Leader. State: Replica: 064166c511684e5498919039d85fab8c, State: Running, Role: LEADER
I20260619 16:45:35.641804 20804 consensus_queue.cc:237] T 00000000000000000000000000000000 P 064166c511684e5498919039d85fab8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "064166c511684e5498919039d85fab8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41833 } }
I20260619 16:45:35.642602 20794 sys_catalog.cc:564] T 00000000000000000000000000000000 P 064166c511684e5498919039d85fab8c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:45:35.643563 20805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 064166c511684e5498919039d85fab8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "064166c511684e5498919039d85fab8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "064166c511684e5498919039d85fab8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41833 } } }
I20260619 16:45:35.643745 20805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 064166c511684e5498919039d85fab8c [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:35.644059 20806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 064166c511684e5498919039d85fab8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 064166c511684e5498919039d85fab8c. Latest consensus state: current_term: 1 leader_uuid: "064166c511684e5498919039d85fab8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "064166c511684e5498919039d85fab8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41833 } } }
I20260619 16:45:35.644169 20806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 064166c511684e5498919039d85fab8c [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:35.645550 20797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:35.640137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48739 (local address 127.0.0.1:41833)
0619 16:45:35.640296 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:35.640302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:35.640448 (+ 146us) server_negotiation.cc:407] Connection header received
0619 16:45:35.640674 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:35.640677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:35.640748 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:35.640860 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:35.641189 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:35.641979 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:35.643143 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:35.643574 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:35.643717 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:35.644832 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:35.644836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:35.644838 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:35.644853 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:35.644883 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:35.644897 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:35.644987 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:35.645366 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 16:45:35.645406 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":11,"server-negotiator.queue_time_us":72}
W20260619 16:45:35.648281 20819 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 064166c511684e5498919039d85fab8c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:45:35.648361 20819 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:45:35.648437 20813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:45:35.649277 20813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:45:35.655062 20813 catalog_manager.cc:1347] Generated new cluster ID: b29bc9d3ab7941269fc90592eec0ce78
I20260619 16:45:35.655151 20813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:45:35.669108 20813 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:45:35.670120 20813 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:45:35.676790 20813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 064166c511684e5498919039d85fab8c: Generated new TSK 0
I20260619 16:45:35.677589 20813 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:45:35.684232 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 20726
W20260619 16:45:35.690265 20711 connection.cc:535] client connection to 127.0.0.1:41833 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41833): Connection reset by peer (error 104)
2026-06-19T16:45:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1453 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:45:35 dist-test-slave-03cd krb5kdc[20825](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:45:35 dist-test-slave-03cd krb5kdc[20825](info): set up 2 sockets
Jun 19 16:45:35 dist-test-slave-03cd krb5kdc[20825](info): commencing operation
krb5kdc: starting...
W20260619 16:45:36.747994 392 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:45:36 dist-test-slave-03cd krb5kdc[20825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:45:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:45:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:45:36.883508 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34091
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:37.001252 20838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:37.001710 20838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:37.001808 20838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:37.006353 20838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:45:37.006493 20838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:37.006536 20838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:37.006600 20838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:45:37.006655 20838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:45:37.012744 20838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34091
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:37.014026 20838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:37.015185 20838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:37.024962 20843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:37.024963 20844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:37.024963 20846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:37.025338 20838 server_base.cc:1034] running on GCE node
I20260619 16:45:37.025969 20838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:37.027233 20838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:37.028475 20838 hybrid_clock.cc:648] HybridClock initialized: now 1781887537028439 us; error 62 us; skew 500 ppm
Jun 19 16:45:37 dist-test-slave-03cd krb5kdc[20825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:37.035833 20838 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:45:37.036942 20838 webserver.cc:460] Webserver started at http://127.0.0.1:39577/ using document root <none> and password file <none>
I20260619 16:45:37.037601 20838 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:37.037706 20838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:37.037954 20838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:37.040372 20838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "9325ca32d40741b9bee5da98929e598c"
format_stamp: "Formatted at 2026-06-19 16:45:37 on dist-test-slave-03cd"
server_key: "904b7ffd86e7f5b4a321405fd40a41a9"
server_key_iv: "bccd19acbcd8ffb31e2ee4b05803857b"
server_key_version: "encryptionkey@0"
I20260619 16:45:37.041174 20838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "9325ca32d40741b9bee5da98929e598c"
format_stamp: "Formatted at 2026-06-19 16:45:37 on dist-test-slave-03cd"
server_key: "904b7ffd86e7f5b4a321405fd40a41a9"
server_key_iv: "bccd19acbcd8ffb31e2ee4b05803857b"
server_key_version: "encryptionkey@0"
I20260619 16:45:37.045601 20838 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:45:37.048454 20853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:37.049952 20838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:45:37.050143 20838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "9325ca32d40741b9bee5da98929e598c"
format_stamp: "Formatted at 2026-06-19 16:45:37 on dist-test-slave-03cd"
server_key: "904b7ffd86e7f5b4a321405fd40a41a9"
server_key_iv: "bccd19acbcd8ffb31e2ee4b05803857b"
server_key_version: "encryptionkey@0"
I20260619 16:45:37.050285 20838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:37.055939 20838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:37.059857 20838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:37.060163 20838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:37.068645 20838 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34091
I20260619 16:45:37.068697 20905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34091 every 8 connection(s)
I20260619 16:45:37.069660 20838 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:45:37.070796 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 20838
I20260619 16:45:37.070945 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:45:37.071270 392 external_mini_cluster.cc:1428] Setting key ba6155d7accddf9e890b6a75fe206b83
I20260619 16:45:37.073292 20906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:37.077533 20909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:37.072858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39461 (local address 127.0.0.1:34091)
0619 16:45:37.073319 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:37.073337 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:37.073375 (+ 38us) server_negotiation.cc:407] Connection header received
0619 16:45:37.074155 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:37.074174 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:37.074497 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:37.074835 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:37.076145 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:37.076151 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:37.076174 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:37.076201 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:37.076263 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:37.076284 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:37.076414 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:37.076513 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 16:45:37.076699 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":164,"threads_started":1}
I20260619 16:45:37.080447 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:45:37.080590 20906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9325ca32d40741b9bee5da98929e598c: Bootstrap starting.
I20260619 16:45:37.083958 20906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9325ca32d40741b9bee5da98929e598c: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:37.084918 20909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:37.083169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55117 (local address 127.0.0.1:34091)
0619 16:45:37.083315 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:37.083321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:37.083472 (+ 151us) server_negotiation.cc:407] Connection header received
0619 16:45:37.083582 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:37.083586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:37.083626 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:37.083772 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:37.084511 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:37.084515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:37.084518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:37.084529 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:37.084559 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:37.084571 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:37.084668 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:37.084757 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 16:45:37.084798 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:45:37.084882 20906 log.cc:824] T 00000000000000000000000000000000 P 9325ca32d40741b9bee5da98929e598c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:37.086911 20906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9325ca32d40741b9bee5da98929e598c: No bootstrap required, opened a new log
I20260619 16:45:37.090603 20906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9325ca32d40741b9bee5da98929e598c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9325ca32d40741b9bee5da98929e598c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34091 } }
I20260619 16:45:37.090916 20906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9325ca32d40741b9bee5da98929e598c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:37.090963 20906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9325ca32d40741b9bee5da98929e598c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9325ca32d40741b9bee5da98929e598c, State: Initialized, Role: FOLLOWER
I20260619 16:45:37.091398 20906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9325ca32d40741b9bee5da98929e598c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9325ca32d40741b9bee5da98929e598c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34091 } }
I20260619 16:45:37.091512 20906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9325ca32d40741b9bee5da98929e598c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:45:37.091553 20906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9325ca32d40741b9bee5da98929e598c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:45:37.091624 20906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9325ca32d40741b9bee5da98929e598c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:37.092684 20906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9325ca32d40741b9bee5da98929e598c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9325ca32d40741b9bee5da98929e598c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34091 } }
I20260619 16:45:37.093003 20906 leader_election.cc:302] T 00000000000000000000000000000000 P 9325ca32d40741b9bee5da98929e598c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9325ca32d40741b9bee5da98929e598c; no voters:
I20260619 16:45:37.093293 20906 leader_election.cc:288] T 00000000000000000000000000000000 P 9325ca32d40741b9bee5da98929e598c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:45:37.093443 20916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9325ca32d40741b9bee5da98929e598c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:37.093724 20916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9325ca32d40741b9bee5da98929e598c [term 1 LEADER]: Becoming Leader. State: Replica: 9325ca32d40741b9bee5da98929e598c, State: Running, Role: LEADER
I20260619 16:45:37.094043 20916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9325ca32d40741b9bee5da98929e598c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9325ca32d40741b9bee5da98929e598c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34091 } }
I20260619 16:45:37.094636 20906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9325ca32d40741b9bee5da98929e598c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:45:37.095813 20917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9325ca32d40741b9bee5da98929e598c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9325ca32d40741b9bee5da98929e598c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9325ca32d40741b9bee5da98929e598c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34091 } } }
I20260619 16:45:37.095963 20917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9325ca32d40741b9bee5da98929e598c [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:37.096328 20918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9325ca32d40741b9bee5da98929e598c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9325ca32d40741b9bee5da98929e598c. Latest consensus state: current_term: 1 leader_uuid: "9325ca32d40741b9bee5da98929e598c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9325ca32d40741b9bee5da98929e598c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34091 } } }
I20260619 16:45:37.096477 20918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9325ca32d40741b9bee5da98929e598c [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:37.096747 20925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:45:37.099611 20925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:45:37.105242 20925 catalog_manager.cc:1347] Generated new cluster ID: e9799139c91242ccb289eb7e0921fa25
I20260619 16:45:37.105333 20925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:45:37.118038 20925 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:45:37.118423 20925 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:45:37.123975 20925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9325ca32d40741b9bee5da98929e598c: Generated new TSK 0
I20260619 16:45:37.124891 20925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:45:37.132393 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 20838
2026-06-19T16:45:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1447 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:45:37 dist-test-slave-03cd krb5kdc[20937](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:45:37 dist-test-slave-03cd krb5kdc[20937](info): set up 2 sockets
Jun 19 16:45:37 dist-test-slave-03cd krb5kdc[20937](info): commencing operation
krb5kdc: starting...
W20260619 16:45:38.228470 392 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:45:38 dist-test-slave-03cd krb5kdc[20937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:45:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:45:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:45:38.380386 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36281
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:38.527088 20950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:38.527486 20950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:38.527577 20950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:38.532140 20950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:45:38.532274 20950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:38.532313 20950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:38.532356 20950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:45:38.532392 20950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:45:38.538622 20950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:38.540217 20950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:38.541769 20950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:38.552686 20958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:38.552769 20955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:38.552769 20956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:38.553331 20950 server_base.cc:1034] running on GCE node
I20260619 16:45:38.554044 20950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:38.555317 20950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:38.556510 20950 hybrid_clock.cc:648] HybridClock initialized: now 1781887538556503 us; error 82 us; skew 500 ppm
Jun 19 16:45:38 dist-test-slave-03cd krb5kdc[20937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:38.559926 20950 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:45:38.561017 20950 webserver.cc:460] Webserver started at http://127.0.0.1:34901/ using document root <none> and password file <none>
I20260619 16:45:38.561695 20950 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:38.561797 20950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:38.562075 20950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:38.564430 20950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "252d86537de84d84a554093b78524ae7"
format_stamp: "Formatted at 2026-06-19 16:45:38 on dist-test-slave-03cd"
server_key: "ad03c6f253a5a1cc728a06409dceb7bd"
server_key_iv: "91fb6771b224c69b016c8e4e8b582c5b"
server_key_version: "encryptionkey@0"
I20260619 16:45:38.565054 20950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "252d86537de84d84a554093b78524ae7"
format_stamp: "Formatted at 2026-06-19 16:45:38 on dist-test-slave-03cd"
server_key: "ad03c6f253a5a1cc728a06409dceb7bd"
server_key_iv: "91fb6771b224c69b016c8e4e8b582c5b"
server_key_version: "encryptionkey@0"
I20260619 16:45:38.569486 20950 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:45:38.572772 20965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:38.574395 20950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:45:38.574608 20950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "252d86537de84d84a554093b78524ae7"
format_stamp: "Formatted at 2026-06-19 16:45:38 on dist-test-slave-03cd"
server_key: "ad03c6f253a5a1cc728a06409dceb7bd"
server_key_iv: "91fb6771b224c69b016c8e4e8b582c5b"
server_key_version: "encryptionkey@0"
I20260619 16:45:38.574762 20950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:38.595301 20950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:38.599047 20950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:38.599296 20950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:38.607312 20950 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36281
I20260619 16:45:38.607385 21017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36281 every 8 connection(s)
I20260619 16:45:38.608502 20950 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:45:38.610060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 20950
I20260619 16:45:38.610265 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:45:38.610551 392 external_mini_cluster.cc:1428] Setting key 8729ecd8798f8be658a02c6ab7e49d97
I20260619 16:45:38.612718 21018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:38.621769 21018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 252d86537de84d84a554093b78524ae7: Bootstrap starting.
Jun 19 16:45:38 dist-test-slave-03cd krb5kdc[20937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 16:45:38.624876 21018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 252d86537de84d84a554093b78524ae7: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:38.625801 21018 log.cc:824] T 00000000000000000000000000000000 P 252d86537de84d84a554093b78524ae7: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:38.628507 21018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 252d86537de84d84a554093b78524ae7: No bootstrap required, opened a new log
I20260619 16:45:38.628799 21021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:38.612451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35717 (local address 127.0.0.1:36281)
0619 16:45:38.612943 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:38.612971 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:38.613017 (+ 46us) server_negotiation.cc:407] Connection header received
0619 16:45:38.613738 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:38.613757 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:38.614185 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:38.614543 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:38.615831 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:38.617306 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:38.618796 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:38.619369 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:38.619517 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:38.623197 (+ 3680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:38.623204 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:38.623223 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:38.623265 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:38.625335 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:38.625882 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:38.625888 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:38.625893 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:38.626075 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:38.626393 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:38.626396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:38.626398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:38.626873 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:38.627099 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:38.627372 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 16:45:38.627776 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":186,"threads_started":1}
I20260619 16:45:38.632815 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:45:38.634166 21018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 252d86537de84d84a554093b78524ae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252d86537de84d84a554093b78524ae7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36281 } }
I20260619 16:45:38.634579 21018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 252d86537de84d84a554093b78524ae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:38.634698 21018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 252d86537de84d84a554093b78524ae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 252d86537de84d84a554093b78524ae7, State: Initialized, Role: FOLLOWER
I20260619 16:45:38.635259 21018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 252d86537de84d84a554093b78524ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252d86537de84d84a554093b78524ae7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36281 } }
I20260619 16:45:38.635470 21018 raft_consensus.cc:388] T 00000000000000000000000000000000 P 252d86537de84d84a554093b78524ae7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:45:38.635547 21018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 252d86537de84d84a554093b78524ae7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:45:38.635684 21018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 252d86537de84d84a554093b78524ae7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:38.636855 21018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 252d86537de84d84a554093b78524ae7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252d86537de84d84a554093b78524ae7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36281 } }
I20260619 16:45:38.637261 21018 leader_election.cc:302] T 00000000000000000000000000000000 P 252d86537de84d84a554093b78524ae7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 252d86537de84d84a554093b78524ae7; no voters:
I20260619 16:45:38.637660 21018 leader_election.cc:288] T 00000000000000000000000000000000 P 252d86537de84d84a554093b78524ae7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:45:38.637930 21028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 252d86537de84d84a554093b78524ae7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:38.638216 21028 raft_consensus.cc:686] T 00000000000000000000000000000000 P 252d86537de84d84a554093b78524ae7 [term 1 LEADER]: Becoming Leader. State: Replica: 252d86537de84d84a554093b78524ae7, State: Running, Role: LEADER
I20260619 16:45:38.638590 21028 consensus_queue.cc:237] T 00000000000000000000000000000000 P 252d86537de84d84a554093b78524ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252d86537de84d84a554093b78524ae7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36281 } }
I20260619 16:45:38.639107 21018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 252d86537de84d84a554093b78524ae7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:45:38.640424 21029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 252d86537de84d84a554093b78524ae7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "252d86537de84d84a554093b78524ae7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252d86537de84d84a554093b78524ae7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36281 } } }
I20260619 16:45:38.640623 21029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 252d86537de84d84a554093b78524ae7 [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:38.640906 21029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 252d86537de84d84a554093b78524ae7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 252d86537de84d84a554093b78524ae7. Latest consensus state: current_term: 1 leader_uuid: "252d86537de84d84a554093b78524ae7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252d86537de84d84a554093b78524ae7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36281 } } }
I20260619 16:45:38.641065 21029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 252d86537de84d84a554093b78524ae7 [sys.catalog]: This master's current role is: LEADER
W20260619 16:45:38.644356 21043 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 252d86537de84d84a554093b78524ae7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:45:38.644456 21043 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:45:38.644559 21037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:45:38.644694 21021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:38.635026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41941 (local address 127.0.0.1:36281)
0619 16:45:38.635163 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:38.635167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:38.635419 (+ 252us) server_negotiation.cc:407] Connection header received
0619 16:45:38.635568 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:38.635573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:38.635622 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:38.635733 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:38.636517 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:38.637195 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:38.638413 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:38.638820 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:38.638933 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:38.640756 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:38.640760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:38.640762 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:38.640790 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:38.642621 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:38.643192 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:38.643195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:38.643197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:38.643280 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:38.643589 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:38.643592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:38.643594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:38.643851 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:38.643954 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:38.644343 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 16:45:38.644473 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:45:38.645385 21037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:45:38.651180 21037 catalog_manager.cc:1347] Generated new cluster ID: 2f9581a5e1ea44439cb127a5f74393db
I20260619 16:45:38.651263 21037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:45:38.660951 21037 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:45:38.662279 21037 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:45:38.668802 21037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 252d86537de84d84a554093b78524ae7: Generated new TSK 0
I20260619 16:45:38.669588 21037 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:45:38.674741 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 20950
2026-06-19T16:45:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1542 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:45:38 dist-test-slave-03cd krb5kdc[21049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:45:38 dist-test-slave-03cd krb5kdc[21049](info): set up 2 sockets
Jun 19 16:45:38 dist-test-slave-03cd krb5kdc[21049](info): commencing operation
krb5kdc: starting...
W20260619 16:45:39.735954 392 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:45:39 dist-test-slave-03cd krb5kdc[21049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:45:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:45:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:45:39.866526 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38925
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:39.988847 21062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:39.989187 21062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:39.989231 21062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:39.993649 21062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:45:39.993733 21062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:39.993753 21062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:39.993772 21062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:45:39.993819 21062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:45:39.999593 21062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:40.000795 21062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:40.001844 21062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:40.013278 21067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:40.013017 21068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:40.013017 21070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:40.013864 21062 server_base.cc:1034] running on GCE node
I20260619 16:45:40.014637 21062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:40.016268 21062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:40.017549 21062 hybrid_clock.cc:648] HybridClock initialized: now 1781887540017483 us; error 90 us; skew 500 ppm
Jun 19 16:45:40 dist-test-slave-03cd krb5kdc[21049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:40.021067 21062 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:45:40.022320 21062 webserver.cc:460] Webserver started at http://127.0.0.1:41533/ using document root <none> and password file <none>
I20260619 16:45:40.023030 21062 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:40.023135 21062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:40.023391 21062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:40.025733 21062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "3864281b18c2495f8214f9fd9cb128ac"
format_stamp: "Formatted at 2026-06-19 16:45:40 on dist-test-slave-03cd"
server_key: "0452380f3c51e1207b12181bf1ba2a2c"
server_key_iv: "a344b7c5a86058b8b2899a9690914baf"
server_key_version: "encryptionkey@0"
I20260619 16:45:40.026358 21062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "3864281b18c2495f8214f9fd9cb128ac"
format_stamp: "Formatted at 2026-06-19 16:45:40 on dist-test-slave-03cd"
server_key: "0452380f3c51e1207b12181bf1ba2a2c"
server_key_iv: "a344b7c5a86058b8b2899a9690914baf"
server_key_version: "encryptionkey@0"
I20260619 16:45:40.030781 21062 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:45:40.033937 21077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:40.035404 21062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 16:45:40.035578 21062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "3864281b18c2495f8214f9fd9cb128ac"
format_stamp: "Formatted at 2026-06-19 16:45:40 on dist-test-slave-03cd"
server_key: "0452380f3c51e1207b12181bf1ba2a2c"
server_key_iv: "a344b7c5a86058b8b2899a9690914baf"
server_key_version: "encryptionkey@0"
I20260619 16:45:40.035775 21062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:40.049711 21062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:40.053401 21062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:40.053750 21062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:40.061941 21062 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38925
I20260619 16:45:40.061980 21129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38925 every 8 connection(s)
I20260619 16:45:40.063052 21062 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:45:40.063870 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 21062
I20260619 16:45:40.064010 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:45:40.064345 392 external_mini_cluster.cc:1428] Setting key 2e781225167bcb0a51383231db900006
I20260619 16:45:40.066664 21130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:40.072659 21130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3864281b18c2495f8214f9fd9cb128ac: Bootstrap starting.
I20260619 16:45:40.074239 21133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:40.065932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41515 (local address 127.0.0.1:38925)
0619 16:45:40.066425 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:40.066440 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:40.066478 (+ 38us) server_negotiation.cc:407] Connection header received
0619 16:45:40.067327 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:40.067345 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:40.067782 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:40.068129 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:40.068539 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:40.069527 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:40.070471 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:40.070977 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:40.071153 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:40.072241 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:40.072247 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:40.072262 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:40.072790 (+ 528us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:40.072871 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:40.072899 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:40.073021 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:40.073194 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 16:45:40.073438 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":152,"threads_started":1}
I20260619 16:45:40.075664 21130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3864281b18c2495f8214f9fd9cb128ac: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:40.076673 21130 log.cc:824] T 00000000000000000000000000000000 P 3864281b18c2495f8214f9fd9cb128ac: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:40.077961 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:45:40.079454 21130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3864281b18c2495f8214f9fd9cb128ac: No bootstrap required, opened a new log
I20260619 16:45:40.084802 21133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:40.080246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44451 (local address 127.0.0.1:38925)
0619 16:45:40.080386 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:40.080391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:40.080560 (+ 169us) server_negotiation.cc:407] Connection header received
0619 16:45:40.080694 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:40.080696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:40.080737 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:40.080806 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:40.081149 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:40.081698 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:40.082974 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:40.083227 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:40.083327 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:40.084284 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:40.084287 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:40.084289 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:40.084303 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:40.084332 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:40.084343 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:40.084403 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:40.084518 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 16:45:40.084574 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:45:40.084579 21130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3864281b18c2495f8214f9fd9cb128ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3864281b18c2495f8214f9fd9cb128ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38925 } }
I20260619 16:45:40.084967 21130 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3864281b18c2495f8214f9fd9cb128ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:40.085031 21130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3864281b18c2495f8214f9fd9cb128ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3864281b18c2495f8214f9fd9cb128ac, State: Initialized, Role: FOLLOWER
I20260619 16:45:40.085557 21130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3864281b18c2495f8214f9fd9cb128ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3864281b18c2495f8214f9fd9cb128ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38925 } }
I20260619 16:45:40.085723 21130 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3864281b18c2495f8214f9fd9cb128ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:45:40.085793 21130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3864281b18c2495f8214f9fd9cb128ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:45:40.085907 21130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3864281b18c2495f8214f9fd9cb128ac [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:40.086963 21130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3864281b18c2495f8214f9fd9cb128ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3864281b18c2495f8214f9fd9cb128ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38925 } }
I20260619 16:45:40.087350 21130 leader_election.cc:302] T 00000000000000000000000000000000 P 3864281b18c2495f8214f9fd9cb128ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3864281b18c2495f8214f9fd9cb128ac; no voters:
I20260619 16:45:40.087646 21130 leader_election.cc:288] T 00000000000000000000000000000000 P 3864281b18c2495f8214f9fd9cb128ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:45:40.087858 21140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3864281b18c2495f8214f9fd9cb128ac [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:40.088140 21140 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3864281b18c2495f8214f9fd9cb128ac [term 1 LEADER]: Becoming Leader. State: Replica: 3864281b18c2495f8214f9fd9cb128ac, State: Running, Role: LEADER
I20260619 16:45:40.088495 21140 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3864281b18c2495f8214f9fd9cb128ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3864281b18c2495f8214f9fd9cb128ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38925 } }
I20260619 16:45:40.089161 21130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3864281b18c2495f8214f9fd9cb128ac [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:45:40.090323 21141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3864281b18c2495f8214f9fd9cb128ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3864281b18c2495f8214f9fd9cb128ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3864281b18c2495f8214f9fd9cb128ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38925 } } }
I20260619 16:45:40.090468 21141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3864281b18c2495f8214f9fd9cb128ac [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:40.090858 21149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:45:40.090804 21142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3864281b18c2495f8214f9fd9cb128ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3864281b18c2495f8214f9fd9cb128ac. Latest consensus state: current_term: 1 leader_uuid: "3864281b18c2495f8214f9fd9cb128ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3864281b18c2495f8214f9fd9cb128ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38925 } } }
I20260619 16:45:40.091171 21142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3864281b18c2495f8214f9fd9cb128ac [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:40.094540 21149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:45:40.100706 21149 catalog_manager.cc:1347] Generated new cluster ID: c706697cf9a9408199bddb1abaf0edaa
I20260619 16:45:40.100803 21149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:45:40.113612 21149 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:45:40.114959 21149 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:45:40.138166 21149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3864281b18c2495f8214f9fd9cb128ac: Generated new TSK 0
I20260619 16:45:40.139071 21149 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:45:40.145203 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 21062
2026-06-19T16:45:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1469 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:45:40 dist-test-slave-03cd krb5kdc[21161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:45:40 dist-test-slave-03cd krb5kdc[21161](info): set up 2 sockets
Jun 19 16:45:40 dist-test-slave-03cd krb5kdc[21161](info): commencing operation
krb5kdc: starting...
W20260619 16:45:41.224220 392 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:45:41 dist-test-slave-03cd krb5kdc[21161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:45:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:45:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:45:41.354645 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41079
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:41.471179 21174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:41.471472 21174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:41.471516 21174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:41.475744 21174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:45:41.475827 21174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:41.475847 21174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:41.475863 21174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:45:41.475879 21174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:45:41.481639 21174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:41.482858 21174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:41.483973 21174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:41.493084 21180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:41.493295 21174 server_base.cc:1034] running on GCE node
W20260619 16:45:41.493130 21182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:41.493081 21179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:41.494032 21174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:41.495218 21174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:41.496378 21174 hybrid_clock.cc:648] HybridClock initialized: now 1781887541496375 us; error 70 us; skew 500 ppm
Jun 19 16:45:41 dist-test-slave-03cd krb5kdc[21161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:41.499279 21174 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:45:41.500327 21174 webserver.cc:460] Webserver started at http://127.0.0.1:35983/ using document root <none> and password file <none>
I20260619 16:45:41.500916 21174 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:41.501008 21174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:41.501252 21174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:41.503438 21174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "568c0354d35f4775b857ca78d968286e"
format_stamp: "Formatted at 2026-06-19 16:45:41 on dist-test-slave-03cd"
server_key: "e0a6453be86f462457dce7441ef9012f"
server_key_iv: "240b657b0498ccdc8fef0fd2d45df90a"
server_key_version: "encryptionkey@0"
I20260619 16:45:41.504124 21174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "568c0354d35f4775b857ca78d968286e"
format_stamp: "Formatted at 2026-06-19 16:45:41 on dist-test-slave-03cd"
server_key: "e0a6453be86f462457dce7441ef9012f"
server_key_iv: "240b657b0498ccdc8fef0fd2d45df90a"
server_key_version: "encryptionkey@0"
I20260619 16:45:41.508208 21174 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:45:41.510910 21189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:41.512413 21174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:45:41.512607 21174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "568c0354d35f4775b857ca78d968286e"
format_stamp: "Formatted at 2026-06-19 16:45:41 on dist-test-slave-03cd"
server_key: "e0a6453be86f462457dce7441ef9012f"
server_key_iv: "240b657b0498ccdc8fef0fd2d45df90a"
server_key_version: "encryptionkey@0"
I20260619 16:45:41.512758 21174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:41.518000 21174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:41.521579 21174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:41.521837 21174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:41.529846 21174 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41079
I20260619 16:45:41.529872 21241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41079 every 8 connection(s)
I20260619 16:45:41.530896 21174 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:45:41.531857 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 21174
I20260619 16:45:41.532006 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:45:41.532300 392 external_mini_cluster.cc:1428] Setting key ca8c6f11c2456c0e7df6cd6e34d32b05
I20260619 16:45:41.534391 21242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:41.537739 21245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:41.533722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55251 (local address 127.0.0.1:41079)
0619 16:45:41.534163 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:41.534179 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:41.534216 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:45:41.534931 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:41.534949 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:41.535253 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:41.535550 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:41.536500 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:41.536505 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:41.536517 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:41.536528 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:41.536585 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:41.536605 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:41.536767 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:41.536838 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 16:45:41.536999 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":134,"threads_started":1}
I20260619 16:45:41.540663 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:45:41.540807 21242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 568c0354d35f4775b857ca78d968286e: Bootstrap starting.
I20260619 16:45:41.543331 21242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 568c0354d35f4775b857ca78d968286e: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:41.544487 21242 log.cc:824] T 00000000000000000000000000000000 P 568c0354d35f4775b857ca78d968286e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:41.544751 21245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:41.542799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55605 (local address 127.0.0.1:41079)
0619 16:45:41.542926 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:41.542931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:41.543078 (+ 147us) server_negotiation.cc:407] Connection header received
0619 16:45:41.543226 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:41.543229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:41.543263 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:41.543322 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:41.544163 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:41.544166 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:41.544168 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:41.544180 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:41.544206 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:41.544216 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:41.544298 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:41.544455 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 16:45:41.544490 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:45:41.546770 21242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 568c0354d35f4775b857ca78d968286e: No bootstrap required, opened a new log
I20260619 16:45:41.550498 21242 raft_consensus.cc:348] T 00000000000000000000000000000000 P 568c0354d35f4775b857ca78d968286e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "568c0354d35f4775b857ca78d968286e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41079 } }
I20260619 16:45:41.550846 21242 raft_consensus.cc:374] T 00000000000000000000000000000000 P 568c0354d35f4775b857ca78d968286e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:41.550918 21242 raft_consensus.cc:729] T 00000000000000000000000000000000 P 568c0354d35f4775b857ca78d968286e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 568c0354d35f4775b857ca78d968286e, State: Initialized, Role: FOLLOWER
I20260619 16:45:41.551344 21242 consensus_queue.cc:260] T 00000000000000000000000000000000 P 568c0354d35f4775b857ca78d968286e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "568c0354d35f4775b857ca78d968286e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41079 } }
I20260619 16:45:41.551498 21242 raft_consensus.cc:388] T 00000000000000000000000000000000 P 568c0354d35f4775b857ca78d968286e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:45:41.551568 21242 raft_consensus.cc:482] T 00000000000000000000000000000000 P 568c0354d35f4775b857ca78d968286e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:45:41.551692 21242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 568c0354d35f4775b857ca78d968286e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:41.552661 21242 raft_consensus.cc:504] T 00000000000000000000000000000000 P 568c0354d35f4775b857ca78d968286e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "568c0354d35f4775b857ca78d968286e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41079 } }
I20260619 16:45:41.553016 21242 leader_election.cc:302] T 00000000000000000000000000000000 P 568c0354d35f4775b857ca78d968286e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 568c0354d35f4775b857ca78d968286e; no voters:
I20260619 16:45:41.553359 21242 leader_election.cc:288] T 00000000000000000000000000000000 P 568c0354d35f4775b857ca78d968286e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:45:41.553495 21252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 568c0354d35f4775b857ca78d968286e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:41.553766 21252 raft_consensus.cc:686] T 00000000000000000000000000000000 P 568c0354d35f4775b857ca78d968286e [term 1 LEADER]: Becoming Leader. State: Replica: 568c0354d35f4775b857ca78d968286e, State: Running, Role: LEADER
I20260619 16:45:41.554169 21252 consensus_queue.cc:237] T 00000000000000000000000000000000 P 568c0354d35f4775b857ca78d968286e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "568c0354d35f4775b857ca78d968286e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41079 } }
I20260619 16:45:41.554745 21242 sys_catalog.cc:564] T 00000000000000000000000000000000 P 568c0354d35f4775b857ca78d968286e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:45:41.556109 21254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 568c0354d35f4775b857ca78d968286e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 568c0354d35f4775b857ca78d968286e. Latest consensus state: current_term: 1 leader_uuid: "568c0354d35f4775b857ca78d968286e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "568c0354d35f4775b857ca78d968286e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41079 } } }
I20260619 16:45:41.556172 21253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 568c0354d35f4775b857ca78d968286e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "568c0354d35f4775b857ca78d968286e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "568c0354d35f4775b857ca78d968286e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41079 } } }
I20260619 16:45:41.556316 21253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 568c0354d35f4775b857ca78d968286e [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:41.556311 21254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 568c0354d35f4775b857ca78d968286e [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:41.556679 21263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:45:41.559909 21263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:45:41.565920 21263 catalog_manager.cc:1347] Generated new cluster ID: 9d1a651f45da420da4f98c900b19b1c3
I20260619 16:45:41.566051 21263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:45:41.594029 21263 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:45:41.594398 21263 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:45:41.602018 21263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 568c0354d35f4775b857ca78d968286e: Generated new TSK 0
I20260619 16:45:41.602763 21263 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:45:41.610823 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 21174
2026-06-19T16:45:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1466 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:45:41 dist-test-slave-03cd krb5kdc[21273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:45:41 dist-test-slave-03cd krb5kdc[21273](info): set up 2 sockets
Jun 19 16:45:41 dist-test-slave-03cd krb5kdc[21273](info): commencing operation
krb5kdc: starting...
W20260619 16:45:42.664112 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:45:42 dist-test-slave-03cd krb5kdc[21273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:45:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:45:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:45:42.804534 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37249
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:42.933785 21286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:42.934149 21286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:42.934350 21286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:42.939572 21286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:45:42.939714 21286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:42.939783 21286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:42.939821 21286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:45:42.939857 21286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:45:42.946493 21286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:42.947904 21286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:42.949031 21286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:42.958863 21294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:42.958863 21291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:42.959252 21292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:42.959245 21286 server_base.cc:1034] running on GCE node
I20260619 16:45:42.960076 21286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:42.961499 21286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:42.962751 21286 hybrid_clock.cc:648] HybridClock initialized: now 1781887542962726 us; error 43 us; skew 500 ppm
Jun 19 16:45:42 dist-test-slave-03cd krb5kdc[21273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:42.969928 21286 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:45:42.971091 21286 webserver.cc:460] Webserver started at http://127.0.0.1:43169/ using document root <none> and password file <none>
I20260619 16:45:42.971786 21286 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:42.971865 21286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:42.972111 21286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:42.974344 21286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "bca78fed88854f7d9856a540c0bbd481"
format_stamp: "Formatted at 2026-06-19 16:45:42 on dist-test-slave-03cd"
server_key: "ed2aaa3f1fa069e8a9060ee3de230612"
server_key_iv: "30fd49869658859d374c6bc239c09ce5"
server_key_version: "encryptionkey@0"
I20260619 16:45:42.975059 21286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "bca78fed88854f7d9856a540c0bbd481"
format_stamp: "Formatted at 2026-06-19 16:45:42 on dist-test-slave-03cd"
server_key: "ed2aaa3f1fa069e8a9060ee3de230612"
server_key_iv: "30fd49869658859d374c6bc239c09ce5"
server_key_version: "encryptionkey@0"
I20260619 16:45:42.979537 21286 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 16:45:42.983465 21301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:42.985409 21286 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:45:42.985611 21286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "bca78fed88854f7d9856a540c0bbd481"
format_stamp: "Formatted at 2026-06-19 16:45:42 on dist-test-slave-03cd"
server_key: "ed2aaa3f1fa069e8a9060ee3de230612"
server_key_iv: "30fd49869658859d374c6bc239c09ce5"
server_key_version: "encryptionkey@0"
I20260619 16:45:42.985795 21286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:43.000185 21286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:43.004602 21286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:43.004838 21286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:43.012780 21286 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37249
I20260619 16:45:43.012799 21353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37249 every 8 connection(s)
I20260619 16:45:43.013867 21286 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:45:43.016960 21354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:43.023161 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 21286
I20260619 16:45:43.023257 21354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bca78fed88854f7d9856a540c0bbd481: Bootstrap starting.
I20260619 16:45:43.023365 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:45:43.023645 392 external_mini_cluster.cc:1428] Setting key c7008015358a43c2832c24c9f4092c38
I20260619 16:45:43.026206 21354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bca78fed88854f7d9856a540c0bbd481: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:43.027037 21354 log.cc:824] T 00000000000000000000000000000000 P bca78fed88854f7d9856a540c0bbd481: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:43.029381 21354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bca78fed88854f7d9856a540c0bbd481: No bootstrap required, opened a new log
Jun 19 16:45:43 dist-test-slave-03cd krb5kdc[21273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 16:45:43.034308 21354 raft_consensus.cc:348] T 00000000000000000000000000000000 P bca78fed88854f7d9856a540c0bbd481 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca78fed88854f7d9856a540c0bbd481" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37249 } }
I20260619 16:45:43.034886 21354 raft_consensus.cc:374] T 00000000000000000000000000000000 P bca78fed88854f7d9856a540c0bbd481 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:43.034978 21354 raft_consensus.cc:729] T 00000000000000000000000000000000 P bca78fed88854f7d9856a540c0bbd481 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bca78fed88854f7d9856a540c0bbd481, State: Initialized, Role: FOLLOWER
I20260619 16:45:43.035557 21354 consensus_queue.cc:260] T 00000000000000000000000000000000 P bca78fed88854f7d9856a540c0bbd481 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca78fed88854f7d9856a540c0bbd481" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37249 } }
I20260619 16:45:43.035825 21354 raft_consensus.cc:388] T 00000000000000000000000000000000 P bca78fed88854f7d9856a540c0bbd481 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:45:43.035905 21354 raft_consensus.cc:482] T 00000000000000000000000000000000 P bca78fed88854f7d9856a540c0bbd481 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:45:43.036022 21354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bca78fed88854f7d9856a540c0bbd481 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:43.037443 21354 raft_consensus.cc:504] T 00000000000000000000000000000000 P bca78fed88854f7d9856a540c0bbd481 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca78fed88854f7d9856a540c0bbd481" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37249 } }
I20260619 16:45:43.037853 21354 leader_election.cc:302] T 00000000000000000000000000000000 P bca78fed88854f7d9856a540c0bbd481 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bca78fed88854f7d9856a540c0bbd481; no voters:
I20260619 16:45:43.038229 21354 leader_election.cc:288] T 00000000000000000000000000000000 P bca78fed88854f7d9856a540c0bbd481 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:45:43.038357 21359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bca78fed88854f7d9856a540c0bbd481 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:43.038607 21359 raft_consensus.cc:686] T 00000000000000000000000000000000 P bca78fed88854f7d9856a540c0bbd481 [term 1 LEADER]: Becoming Leader. State: Replica: bca78fed88854f7d9856a540c0bbd481, State: Running, Role: LEADER
I20260619 16:45:43.038997 21359 consensus_queue.cc:237] T 00000000000000000000000000000000 P bca78fed88854f7d9856a540c0bbd481 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca78fed88854f7d9856a540c0bbd481" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37249 } }
I20260619 16:45:43.039497 21354 sys_catalog.cc:564] T 00000000000000000000000000000000 P bca78fed88854f7d9856a540c0bbd481 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:45:43.040458 21361 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca78fed88854f7d9856a540c0bbd481 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bca78fed88854f7d9856a540c0bbd481. Latest consensus state: current_term: 1 leader_uuid: "bca78fed88854f7d9856a540c0bbd481" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca78fed88854f7d9856a540c0bbd481" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37249 } } }
I20260619 16:45:43.040606 21361 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca78fed88854f7d9856a540c0bbd481 [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:43.041839 21360 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca78fed88854f7d9856a540c0bbd481 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bca78fed88854f7d9856a540c0bbd481" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca78fed88854f7d9856a540c0bbd481" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37249 } } }
I20260619 16:45:43.041951 21360 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca78fed88854f7d9856a540c0bbd481 [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:43.042196 21357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:43.025264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:59761 (local address 127.0.0.1:37249)
0619 16:45:43.025739 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:43.025757 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:43.025795 (+ 38us) server_negotiation.cc:407] Connection header received
0619 16:45:43.026586 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:43.026606 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:43.027004 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:43.027378 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:43.028384 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:43.029417 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:43.030623 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:43.031143 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:43.031283 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:43.034587 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:43.034594 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:43.034613 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:43.034895 (+ 282us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:43.037387 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:43.038157 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:43.038166 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:43.038175 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:43.038325 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:43.038759 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:43.038762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:43.038764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:43.039354 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:43.039594 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:43.040655 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0619 16:45:43.040963 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":169,"threads_started":1}
I20260619 16:45:43.042820 21367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:45:43.045639 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:45:43.046032 21367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:45:43.052505 21367 catalog_manager.cc:1347] Generated new cluster ID: 5b0ed5fb50234b4aa7650f1599d999e1
I20260619 16:45:43.052621 21367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:45:43.058390 21357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:43.048620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:46833 (local address 127.0.0.1:37249)
0619 16:45:43.048747 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:43.048752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:43.049072 (+ 320us) server_negotiation.cc:407] Connection header received
0619 16:45:43.049180 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:43.049183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:43.049242 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:43.049329 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:43.050246 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:43.050890 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:43.051831 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:43.052086 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:43.052212 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:43.054278 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:43.054282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:43.054285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:43.054318 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:43.056171 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:43.056779 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:43.056786 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:43.056788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:43.056883 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:43.057239 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:43.057242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:43.057244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:43.057474 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:43.057622 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:43.057915 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 16:45:43.058210 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 16:45:43.073763 21367 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:45:43.074786 21367 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:45:43.082414 21367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bca78fed88854f7d9856a540c0bbd481: Generated new TSK 0
I20260619 16:45:43.083354 21367 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:45:43.095163 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 21286
W20260619 16:45:43.100630 21271 connection.cc:535] client connection to 127.0.0.1:37249 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37249): Connection reset by peer (error 104)
2026-06-19T16:45:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1483 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:45:43 dist-test-slave-03cd krb5kdc[21385](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:45:43 dist-test-slave-03cd krb5kdc[21385](info): set up 2 sockets
Jun 19 16:45:43 dist-test-slave-03cd krb5kdc[21385](info): commencing operation
krb5kdc: starting...
W20260619 16:45:44.160110 392 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:45:44 dist-test-slave-03cd krb5kdc[21385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:45:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:45:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:45:44.295517 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39119
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:44.412766 21398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:44.413141 21398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:44.413229 21398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:44.417567 21398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:45:44.417696 21398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:44.417732 21398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:44.417773 21398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:45:44.417807 21398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:45:44.423483 21398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:44.424747 21398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:44.425812 21398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:44.435598 21406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:44.435616 21404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:44.435608 21403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:44.436172 21398 server_base.cc:1034] running on GCE node
I20260619 16:45:44.436888 21398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:44.438165 21398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:44.439378 21398 hybrid_clock.cc:648] HybridClock initialized: now 1781887544439361 us; error 46 us; skew 500 ppm
Jun 19 16:45:44 dist-test-slave-03cd krb5kdc[21385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:44.442550 21398 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:45:44.443560 21398 webserver.cc:460] Webserver started at http://127.0.0.1:41675/ using document root <none> and password file <none>
I20260619 16:45:44.444279 21398 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:44.444346 21398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:44.444523 21398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:44.446602 21398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "885ce2a2a59f45808b606bc7fc06f8fd"
format_stamp: "Formatted at 2026-06-19 16:45:44 on dist-test-slave-03cd"
server_key: "43b72d34588b8a3a08b645ced31698d6"
server_key_iv: "971941bdcdb4b948138fd1dadb765a43"
server_key_version: "encryptionkey@0"
I20260619 16:45:44.447150 21398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "885ce2a2a59f45808b606bc7fc06f8fd"
format_stamp: "Formatted at 2026-06-19 16:45:44 on dist-test-slave-03cd"
server_key: "43b72d34588b8a3a08b645ced31698d6"
server_key_iv: "971941bdcdb4b948138fd1dadb765a43"
server_key_version: "encryptionkey@0"
I20260619 16:45:44.450978 21398 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:45:44.453490 21413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:44.454749 21398 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:45:44.454891 21398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "885ce2a2a59f45808b606bc7fc06f8fd"
format_stamp: "Formatted at 2026-06-19 16:45:44 on dist-test-slave-03cd"
server_key: "43b72d34588b8a3a08b645ced31698d6"
server_key_iv: "971941bdcdb4b948138fd1dadb765a43"
server_key_version: "encryptionkey@0"
I20260619 16:45:44.455011 21398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:44.465878 21398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:44.469285 21398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:44.469504 21398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:44.478287 21398 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39119
I20260619 16:45:44.478307 21465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39119 every 8 connection(s)
I20260619 16:45:44.479346 21398 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:45:44.482302 21466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:44.482759 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 21398
I20260619 16:45:44.482904 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:45:44.483230 392 external_mini_cluster.cc:1428] Setting key 699d071e72a1a010229c6fe4f93cb2fc
I20260619 16:45:44.489354 21466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 885ce2a2a59f45808b606bc7fc06f8fd: Bootstrap starting.
I20260619 16:45:44.491912 21466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 885ce2a2a59f45808b606bc7fc06f8fd: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:44.492641 21469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:44.484797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:53301 (local address 127.0.0.1:39119)
0619 16:45:44.485258 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:44.485273 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:44.485308 (+ 35us) server_negotiation.cc:407] Connection header received
0619 16:45:44.485993 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:44.486011 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:44.486363 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:44.486720 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:44.487129 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:44.488160 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:44.489086 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:44.489640 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:44.489812 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:44.490854 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:44.490860 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:44.490873 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:44.491282 (+ 409us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:44.491340 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:44.491363 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:44.491510 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:44.491731 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 16:45:44.491879 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":139,"threads_started":1}
I20260619 16:45:44.492667 21466 log.cc:824] T 00000000000000000000000000000000 P 885ce2a2a59f45808b606bc7fc06f8fd: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:44.494980 21466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 885ce2a2a59f45808b606bc7fc06f8fd: No bootstrap required, opened a new log
I20260619 16:45:44.495927 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:45:44.499912 21466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 885ce2a2a59f45808b606bc7fc06f8fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885ce2a2a59f45808b606bc7fc06f8fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39119 } }
I20260619 16:45:44.500173 21466 raft_consensus.cc:374] T 00000000000000000000000000000000 P 885ce2a2a59f45808b606bc7fc06f8fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:44.500209 21466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 885ce2a2a59f45808b606bc7fc06f8fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 885ce2a2a59f45808b606bc7fc06f8fd, State: Initialized, Role: FOLLOWER
I20260619 16:45:44.500751 21466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 885ce2a2a59f45808b606bc7fc06f8fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885ce2a2a59f45808b606bc7fc06f8fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39119 } }
I20260619 16:45:44.500912 21466 raft_consensus.cc:388] T 00000000000000000000000000000000 P 885ce2a2a59f45808b606bc7fc06f8fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:45:44.500986 21466 raft_consensus.cc:482] T 00000000000000000000000000000000 P 885ce2a2a59f45808b606bc7fc06f8fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:45:44.501101 21466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 885ce2a2a59f45808b606bc7fc06f8fd [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:44.502467 21469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:44.498213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:40461 (local address 127.0.0.1:39119)
0619 16:45:44.498366 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:44.498372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:44.498468 (+ 96us) server_negotiation.cc:407] Connection header received
0619 16:45:44.498559 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:44.498562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:44.498609 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:44.498702 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:44.498924 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:44.499720 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:44.500604 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:44.500956 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:44.501076 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:44.501980 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:44.501984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:44.501987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:44.502001 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:44.502038 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:44.502052 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:44.502142 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:44.502251 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 16:45:44.502293 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:45:44.502447 21466 raft_consensus.cc:504] T 00000000000000000000000000000000 P 885ce2a2a59f45808b606bc7fc06f8fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885ce2a2a59f45808b606bc7fc06f8fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39119 } }
I20260619 16:45:44.502835 21466 leader_election.cc:302] T 00000000000000000000000000000000 P 885ce2a2a59f45808b606bc7fc06f8fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 885ce2a2a59f45808b606bc7fc06f8fd; no voters:
I20260619 16:45:44.503221 21466 leader_election.cc:288] T 00000000000000000000000000000000 P 885ce2a2a59f45808b606bc7fc06f8fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:45:44.503298 21476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 885ce2a2a59f45808b606bc7fc06f8fd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:44.503613 21476 raft_consensus.cc:686] T 00000000000000000000000000000000 P 885ce2a2a59f45808b606bc7fc06f8fd [term 1 LEADER]: Becoming Leader. State: Replica: 885ce2a2a59f45808b606bc7fc06f8fd, State: Running, Role: LEADER
I20260619 16:45:44.503949 21476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 885ce2a2a59f45808b606bc7fc06f8fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885ce2a2a59f45808b606bc7fc06f8fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39119 } }
I20260619 16:45:44.504626 21466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 885ce2a2a59f45808b606bc7fc06f8fd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:45:44.505745 21477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 885ce2a2a59f45808b606bc7fc06f8fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 885ce2a2a59f45808b606bc7fc06f8fd. Latest consensus state: current_term: 1 leader_uuid: "885ce2a2a59f45808b606bc7fc06f8fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885ce2a2a59f45808b606bc7fc06f8fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39119 } } }
I20260619 16:45:44.505900 21477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 885ce2a2a59f45808b606bc7fc06f8fd [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:44.506304 21485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:45:44.506322 21478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 885ce2a2a59f45808b606bc7fc06f8fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "885ce2a2a59f45808b606bc7fc06f8fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885ce2a2a59f45808b606bc7fc06f8fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39119 } } }
I20260619 16:45:44.506462 21478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 885ce2a2a59f45808b606bc7fc06f8fd [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:44.509714 21485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:45:44.516135 21485 catalog_manager.cc:1347] Generated new cluster ID: a54829048e58427c81e3669d8e4303dc
I20260619 16:45:44.516237 21485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:45:44.557469 21485 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:45:44.558861 21485 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:45:44.569900 21485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 885ce2a2a59f45808b606bc7fc06f8fd: Generated new TSK 0
I20260619 16:45:44.570775 21485 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:45:44.575485 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 21398
W20260619 16:45:44.581210 21383 connection.cc:535] client connection to 127.0.0.1:39119 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39119): Connection reset by peer (error 104)
2026-06-19T16:45:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1480 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:45:44 dist-test-slave-03cd krb5kdc[21497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:45:44 dist-test-slave-03cd krb5kdc[21497](info): set up 2 sockets
Jun 19 16:45:44 dist-test-slave-03cd krb5kdc[21497](info): commencing operation
krb5kdc: starting...
W20260619 16:45:45.635977 392 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:45:45 dist-test-slave-03cd krb5kdc[21497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:45:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:45:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:45:45.767750 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40523
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:45.882418 21510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:45.882762 21510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:45.882853 21510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:45.887092 21510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:45:45.887214 21510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:45.887248 21510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:45.887290 21510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:45:45.887324 21510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:45:45.893115 21510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:45.894349 21510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:45.895426 21510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:45:45.904874 21510 server_base.cc:1034] running on GCE node
W20260619 16:45:45.904829 21518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:45.904827 21516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:45.904827 21515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:45.905855 21510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:45.907094 21510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:45.908294 21510 hybrid_clock.cc:648] HybridClock initialized: now 1781887545908262 us; error 50 us; skew 500 ppm
Jun 19 16:45:45 dist-test-slave-03cd krb5kdc[21497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887545, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:45.911399 21510 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:45:45.912508 21510 webserver.cc:460] Webserver started at http://127.0.0.1:35851/ using document root <none> and password file <none>
I20260619 16:45:45.913156 21510 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:45.913360 21510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:45.913633 21510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:45.915940 21510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "9ef5650aa40e49dbbf377cc564e9396d"
format_stamp: "Formatted at 2026-06-19 16:45:45 on dist-test-slave-03cd"
server_key: "e6840b0480dddba5f109b06e224ce465"
server_key_iv: "8a9cc651114d8f49055ed30ba060eb3e"
server_key_version: "encryptionkey@0"
I20260619 16:45:45.916571 21510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "9ef5650aa40e49dbbf377cc564e9396d"
format_stamp: "Formatted at 2026-06-19 16:45:45 on dist-test-slave-03cd"
server_key: "e6840b0480dddba5f109b06e224ce465"
server_key_iv: "8a9cc651114d8f49055ed30ba060eb3e"
server_key_version: "encryptionkey@0"
I20260619 16:45:45.920821 21510 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:45:45.923838 21525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:45.925340 21510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:45:45.925478 21510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "9ef5650aa40e49dbbf377cc564e9396d"
format_stamp: "Formatted at 2026-06-19 16:45:45 on dist-test-slave-03cd"
server_key: "e6840b0480dddba5f109b06e224ce465"
server_key_iv: "8a9cc651114d8f49055ed30ba060eb3e"
server_key_version: "encryptionkey@0"
I20260619 16:45:45.925652 21510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:45.931308 21510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:45.935001 21510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:45.935217 21510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:45.942699 21510 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40523
I20260619 16:45:45.942714 21577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40523 every 8 connection(s)
I20260619 16:45:45.943756 21510 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:45:45.944875 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 21510
I20260619 16:45:45.944980 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:45:45.945307 392 external_mini_cluster.cc:1428] Setting key ccae212eaaf7f18fdb239a440866ce4f
I20260619 16:45:45.947196 21578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:45.951201 21581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:45.946771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:33321 (local address 127.0.0.1:40523)
0619 16:45:45.947211 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:45.947232 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:45.947277 (+ 45us) server_negotiation.cc:407] Connection header received
0619 16:45:45.948300 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:45.948321 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:45.948671 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:45.949048 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:45.949796 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:45.949802 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:45.949816 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:45.949828 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:45.949892 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:45.949918 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:45.950078 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:45.950143 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 16:45:45.950296 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":113,"threads_started":1}
I20260619 16:45:45.954116 21578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ef5650aa40e49dbbf377cc564e9396d: Bootstrap starting.
I20260619 16:45:45.954173 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:45:45.956575 21578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ef5650aa40e49dbbf377cc564e9396d: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:45.957541 21578 log.cc:824] T 00000000000000000000000000000000 P 9ef5650aa40e49dbbf377cc564e9396d: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:45.958175 21581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:45.956318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:38417 (local address 127.0.0.1:40523)
0619 16:45:45.956458 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:45.956463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:45.956730 (+ 267us) server_negotiation.cc:407] Connection header received
0619 16:45:45.956820 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:45.956822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:45.956855 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:45.956922 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:45.957678 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:45.957685 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:45.957690 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:45.957702 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:45.957736 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:45.957747 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:45.957817 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:45.957981 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 16:45:45.958024 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:45:45.959810 21578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ef5650aa40e49dbbf377cc564e9396d: No bootstrap required, opened a new log
I20260619 16:45:45.963711 21578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ef5650aa40e49dbbf377cc564e9396d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef5650aa40e49dbbf377cc564e9396d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40523 } }
I20260619 16:45:45.963986 21578 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ef5650aa40e49dbbf377cc564e9396d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:45.964023 21578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ef5650aa40e49dbbf377cc564e9396d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef5650aa40e49dbbf377cc564e9396d, State: Initialized, Role: FOLLOWER
I20260619 16:45:45.964411 21578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ef5650aa40e49dbbf377cc564e9396d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef5650aa40e49dbbf377cc564e9396d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40523 } }
I20260619 16:45:45.964524 21578 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ef5650aa40e49dbbf377cc564e9396d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:45:45.964570 21578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ef5650aa40e49dbbf377cc564e9396d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:45:45.964635 21578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ef5650aa40e49dbbf377cc564e9396d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:45.965571 21578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ef5650aa40e49dbbf377cc564e9396d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef5650aa40e49dbbf377cc564e9396d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40523 } }
I20260619 16:45:45.965878 21578 leader_election.cc:302] T 00000000000000000000000000000000 P 9ef5650aa40e49dbbf377cc564e9396d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ef5650aa40e49dbbf377cc564e9396d; no voters:
I20260619 16:45:45.966169 21578 leader_election.cc:288] T 00000000000000000000000000000000 P 9ef5650aa40e49dbbf377cc564e9396d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:45:45.966380 21588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ef5650aa40e49dbbf377cc564e9396d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:45.966653 21588 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ef5650aa40e49dbbf377cc564e9396d [term 1 LEADER]: Becoming Leader. State: Replica: 9ef5650aa40e49dbbf377cc564e9396d, State: Running, Role: LEADER
I20260619 16:45:45.966993 21588 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ef5650aa40e49dbbf377cc564e9396d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef5650aa40e49dbbf377cc564e9396d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40523 } }
I20260619 16:45:45.967629 21578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ef5650aa40e49dbbf377cc564e9396d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:45:45.968704 21589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef5650aa40e49dbbf377cc564e9396d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ef5650aa40e49dbbf377cc564e9396d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef5650aa40e49dbbf377cc564e9396d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40523 } } }
I20260619 16:45:45.968830 21589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef5650aa40e49dbbf377cc564e9396d [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:45.969274 21597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:45:45.969493 21590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef5650aa40e49dbbf377cc564e9396d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ef5650aa40e49dbbf377cc564e9396d. Latest consensus state: current_term: 1 leader_uuid: "9ef5650aa40e49dbbf377cc564e9396d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef5650aa40e49dbbf377cc564e9396d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40523 } } }
I20260619 16:45:45.969611 21590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef5650aa40e49dbbf377cc564e9396d [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:45.973038 21597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:45:45.979781 21597 catalog_manager.cc:1347] Generated new cluster ID: 3bc6487d6f024551be61e5e56a435926
I20260619 16:45:45.979876 21597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:45:46.023407 21597 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:45:46.023845 21597 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:45:46.037410 21597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ef5650aa40e49dbbf377cc564e9396d: Generated new TSK 0
I20260619 16:45:46.038373 21597 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:45:46.042073 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 21510
2026-06-19T16:45:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1466 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:45:46 dist-test-slave-03cd krb5kdc[21609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:45:46 dist-test-slave-03cd krb5kdc[21609](info): set up 2 sockets
Jun 19 16:45:46 dist-test-slave-03cd krb5kdc[21609](info): commencing operation
krb5kdc: starting...
W20260619 16:45:47.108064 392 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:45:47 dist-test-slave-03cd krb5kdc[21609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:45:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:45:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:45:47.242852 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40939
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:47.363592 21622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:47.363992 21622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:47.364082 21622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:47.368332 21622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:45:47.368461 21622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:47.368501 21622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:47.368541 21622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:45:47.368575 21622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:45:47.374578 21622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40939
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:47.376024 21622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:47.377233 21622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:47.388084 21627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:47.388091 21630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:47.388125 21628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:47.388412 21622 server_base.cc:1034] running on GCE node
I20260619 16:45:47.389083 21622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:47.390450 21622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:47.391731 21622 hybrid_clock.cc:648] HybridClock initialized: now 1781887547391695 us; error 56 us; skew 500 ppm
Jun 19 16:45:47 dist-test-slave-03cd krb5kdc[21609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:47.394757 21622 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:45:47.395871 21622 webserver.cc:460] Webserver started at http://127.0.0.1:37281/ using document root <none> and password file <none>
I20260619 16:45:47.396447 21622 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:47.396509 21622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:47.396695 21622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:47.398874 21622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "3ebd4b8eef5b4c5c933c55c6c6d03115"
format_stamp: "Formatted at 2026-06-19 16:45:47 on dist-test-slave-03cd"
server_key: "bad0c317e26925156c51ce447b47f139"
server_key_iv: "9f2a3d38a05fe8ab18ae78c8562cf2b3"
server_key_version: "encryptionkey@0"
I20260619 16:45:47.399415 21622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "3ebd4b8eef5b4c5c933c55c6c6d03115"
format_stamp: "Formatted at 2026-06-19 16:45:47 on dist-test-slave-03cd"
server_key: "bad0c317e26925156c51ce447b47f139"
server_key_iv: "9f2a3d38a05fe8ab18ae78c8562cf2b3"
server_key_version: "encryptionkey@0"
I20260619 16:45:47.403483 21622 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 16:45:47.406317 21637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:47.407888 21622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:45:47.408025 21622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "3ebd4b8eef5b4c5c933c55c6c6d03115"
format_stamp: "Formatted at 2026-06-19 16:45:47 on dist-test-slave-03cd"
server_key: "bad0c317e26925156c51ce447b47f139"
server_key_iv: "9f2a3d38a05fe8ab18ae78c8562cf2b3"
server_key_version: "encryptionkey@0"
I20260619 16:45:47.408140 21622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:47.421108 21622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:47.424646 21622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:47.424814 21622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:47.432614 21622 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40939
I20260619 16:45:47.432631 21689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40939 every 8 connection(s)
I20260619 16:45:47.433652 21622 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:45:47.436573 21690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:47.440745 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 21622
I20260619 16:45:47.440881 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:45:47.441165 392 external_mini_cluster.cc:1428] Setting key 90fae93dc8430f3f467be46e516ddb13
I20260619 16:45:47.442778 21690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ebd4b8eef5b4c5c933c55c6c6d03115: Bootstrap starting.
I20260619 16:45:47.445231 21690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ebd4b8eef5b4c5c933c55c6c6d03115: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:47.446017 21690 log.cc:824] T 00000000000000000000000000000000 P 3ebd4b8eef5b4c5c933c55c6c6d03115: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:47.448129 21690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ebd4b8eef5b4c5c933c55c6c6d03115: No bootstrap required, opened a new log
Jun 19 16:45:47 dist-test-slave-03cd krb5kdc[21609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 16:45:47.452867 21690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ebd4b8eef5b4c5c933c55c6c6d03115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ebd4b8eef5b4c5c933c55c6c6d03115" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40939 } }
I20260619 16:45:47.453169 21690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ebd4b8eef5b4c5c933c55c6c6d03115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:47.453254 21690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ebd4b8eef5b4c5c933c55c6c6d03115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ebd4b8eef5b4c5c933c55c6c6d03115, State: Initialized, Role: FOLLOWER
I20260619 16:45:47.453799 21690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ebd4b8eef5b4c5c933c55c6c6d03115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ebd4b8eef5b4c5c933c55c6c6d03115" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40939 } }
I20260619 16:45:47.453991 21690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ebd4b8eef5b4c5c933c55c6c6d03115 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:45:47.454066 21690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ebd4b8eef5b4c5c933c55c6c6d03115 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:45:47.454200 21690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ebd4b8eef5b4c5c933c55c6c6d03115 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:47.455492 21690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ebd4b8eef5b4c5c933c55c6c6d03115 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ebd4b8eef5b4c5c933c55c6c6d03115" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40939 } }
I20260619 16:45:47.455931 21690 leader_election.cc:302] T 00000000000000000000000000000000 P 3ebd4b8eef5b4c5c933c55c6c6d03115 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ebd4b8eef5b4c5c933c55c6c6d03115; no voters:
I20260619 16:45:47.456357 21690 leader_election.cc:288] T 00000000000000000000000000000000 P 3ebd4b8eef5b4c5c933c55c6c6d03115 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:45:47.456524 21695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ebd4b8eef5b4c5c933c55c6c6d03115 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:47.456847 21695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ebd4b8eef5b4c5c933c55c6c6d03115 [term 1 LEADER]: Becoming Leader. State: Replica: 3ebd4b8eef5b4c5c933c55c6c6d03115, State: Running, Role: LEADER
I20260619 16:45:47.457202 21695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ebd4b8eef5b4c5c933c55c6c6d03115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ebd4b8eef5b4c5c933c55c6c6d03115" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40939 } }
I20260619 16:45:47.457262 21693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:47.442685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:49763 (local address 127.0.0.1:40939)
0619 16:45:47.443178 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:47.443193 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:47.443231 (+ 38us) server_negotiation.cc:407] Connection header received
0619 16:45:47.444001 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:47.444022 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:47.444395 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:47.444791 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:47.445689 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:47.446749 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:47.447654 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:47.448188 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:47.448363 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:47.451413 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:47.451419 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:47.451435 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:47.451469 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:47.453523 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:47.454204 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:47.454214 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:47.454221 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:47.454372 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:47.454750 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:47.454753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:47.454755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:47.455314 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:47.455533 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:47.455947 (+ 414us) server_negotiation.cc:299] Negotiation successful
0619 16:45:47.456192 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":149,"threads_started":1}
I20260619 16:45:47.457722 21690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ebd4b8eef5b4c5c933c55c6c6d03115 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:45:47.458902 21696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ebd4b8eef5b4c5c933c55c6c6d03115 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ebd4b8eef5b4c5c933c55c6c6d03115" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ebd4b8eef5b4c5c933c55c6c6d03115" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40939 } } }
I20260619 16:45:47.459046 21696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ebd4b8eef5b4c5c933c55c6c6d03115 [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:47.459280 21696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ebd4b8eef5b4c5c933c55c6c6d03115 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ebd4b8eef5b4c5c933c55c6c6d03115. Latest consensus state: current_term: 1 leader_uuid: "3ebd4b8eef5b4c5c933c55c6c6d03115" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ebd4b8eef5b4c5c933c55c6c6d03115" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40939 } } }
I20260619 16:45:47.459378 21696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ebd4b8eef5b4c5c933c55c6c6d03115 [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:47.459546 21704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:45:47.460904 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:45:47.463228 21704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:45:47.470413 21704 catalog_manager.cc:1347] Generated new cluster ID: 21b096374ec9415cb70dddecbea2cf9a
I20260619 16:45:47.470499 21704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:45:47.476625 21693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:47.463146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:53803 (local address 127.0.0.1:40939)
0619 16:45:47.463265 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:47.463271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:47.465763 (+ 2492us) server_negotiation.cc:407] Connection header received
0619 16:45:47.465949 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:47.465952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:47.466015 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:47.466108 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:47.466985 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:47.468051 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:47.469337 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:47.469682 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:47.469805 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:47.472826 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:47.472830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:47.472832 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:47.472862 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:47.474603 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:47.475210 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:47.475219 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:47.475221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:47.475306 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:47.475655 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:47.475693 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:47.475695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:47.475890 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:47.476051 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:47.476332 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 16:45:47.476440 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 16:45:47.484869 21704 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:45:47.486246 21704 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:45:47.496031 21704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ebd4b8eef5b4c5c933c55c6c6d03115: Generated new TSK 0
I20260619 16:45:47.496909 21704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:45:47.507560 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 21622
W20260619 16:45:47.513459 21607 connection.cc:535] client connection to 127.0.0.1:40939 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40939): Connection reset by peer (error 104)
2026-06-19T16:45:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1465 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:45:47 dist-test-slave-03cd krb5kdc[21721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:45:47 dist-test-slave-03cd krb5kdc[21721](info): set up 2 sockets
Jun 19 16:45:47 dist-test-slave-03cd krb5kdc[21721](info): commencing operation
krb5kdc: starting...
W20260619 16:45:48.576539 392 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:45:48 dist-test-slave-03cd krb5kdc[21721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:45:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:45:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:45:48.719936 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43937
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:48.852854 21734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:48.853281 21734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:48.853366 21734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:48.858031 21734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:45:48.858173 21734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:48.858214 21734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:48.858256 21734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:45:48.858294 21734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:45:48.868238 21734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43937
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:48.870064 21734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:48.871438 21734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:48.882432 21742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:48.882592 21734 server_base.cc:1034] running on GCE node
W20260619 16:45:48.882392 21739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:48.882423 21740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:48.883466 21734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:48.884809 21734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:48.886004 21734 hybrid_clock.cc:648] HybridClock initialized: now 1781887548885978 us; error 44 us; skew 500 ppm
Jun 19 16:45:48 dist-test-slave-03cd krb5kdc[21721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:48.893370 21734 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:45:48.894470 21734 webserver.cc:460] Webserver started at http://127.0.0.1:32933/ using document root <none> and password file <none>
I20260619 16:45:48.895092 21734 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:48.895169 21734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:48.895358 21734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:48.898200 21734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "de87b2f9fbf948338905101510a6a269"
format_stamp: "Formatted at 2026-06-19 16:45:48 on dist-test-slave-03cd"
server_key: "32764b857aeaa08d5cc32ad35b67783c"
server_key_iv: "ded27dc646cd2a5f73ea6fc092144106"
server_key_version: "encryptionkey@0"
I20260619 16:45:48.898890 21734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "de87b2f9fbf948338905101510a6a269"
format_stamp: "Formatted at 2026-06-19 16:45:48 on dist-test-slave-03cd"
server_key: "32764b857aeaa08d5cc32ad35b67783c"
server_key_iv: "ded27dc646cd2a5f73ea6fc092144106"
server_key_version: "encryptionkey@0"
I20260619 16:45:48.903543 21734 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:45:48.906654 21749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:48.908378 21734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:45:48.908567 21734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "de87b2f9fbf948338905101510a6a269"
format_stamp: "Formatted at 2026-06-19 16:45:48 on dist-test-slave-03cd"
server_key: "32764b857aeaa08d5cc32ad35b67783c"
server_key_iv: "ded27dc646cd2a5f73ea6fc092144106"
server_key_version: "encryptionkey@0"
I20260619 16:45:48.908743 21734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:48.934062 21734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:48.938053 21734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:48.938297 21734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:48.946486 21734 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43937
I20260619 16:45:48.946502 21801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43937 every 8 connection(s)
I20260619 16:45:48.947602 21734 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:45:48.949081 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 21734
I20260619 16:45:48.949275 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:45:48.949606 392 external_mini_cluster.cc:1428] Setting key 185c61af50c08aa776e900f9714d5216
I20260619 16:45:48.951071 21802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:48.957787 21802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de87b2f9fbf948338905101510a6a269: Bootstrap starting.
I20260619 16:45:48.960065 21805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:48.951217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:56419 (local address 127.0.0.1:43937)
0619 16:45:48.951764 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:48.951781 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:48.951818 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:45:48.952506 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:48.952523 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:48.952892 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:48.953262 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:48.953766 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:48.954783 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:48.955966 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:48.956477 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:48.956664 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:48.957780 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:48.957785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:48.957798 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:48.958299 (+ 501us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:48.958370 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:48.958395 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:48.958561 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:48.958787 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 16:45:48.958941 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":158,"threads_started":1}
I20260619 16:45:48.960801 21802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de87b2f9fbf948338905101510a6a269: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:48.961735 21802 log.cc:824] T 00000000000000000000000000000000 P de87b2f9fbf948338905101510a6a269: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:48.963785 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:45:48.964427 21802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de87b2f9fbf948338905101510a6a269: No bootstrap required, opened a new log
I20260619 16:45:48.968931 21802 raft_consensus.cc:348] T 00000000000000000000000000000000 P de87b2f9fbf948338905101510a6a269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de87b2f9fbf948338905101510a6a269" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43937 } }
I20260619 16:45:48.969466 21802 raft_consensus.cc:374] T 00000000000000000000000000000000 P de87b2f9fbf948338905101510a6a269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:48.969533 21802 raft_consensus.cc:729] T 00000000000000000000000000000000 P de87b2f9fbf948338905101510a6a269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de87b2f9fbf948338905101510a6a269, State: Initialized, Role: FOLLOWER
I20260619 16:45:48.970067 21802 consensus_queue.cc:260] T 00000000000000000000000000000000 P de87b2f9fbf948338905101510a6a269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de87b2f9fbf948338905101510a6a269" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43937 } }
I20260619 16:45:48.970237 21802 raft_consensus.cc:388] T 00000000000000000000000000000000 P de87b2f9fbf948338905101510a6a269 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:45:48.970332 21802 raft_consensus.cc:482] T 00000000000000000000000000000000 P de87b2f9fbf948338905101510a6a269 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:45:48.970432 21802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de87b2f9fbf948338905101510a6a269 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:48.971582 21805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:48.966212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:35895 (local address 127.0.0.1:43937)
0619 16:45:48.966364 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:48.966372 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:48.966609 (+ 237us) server_negotiation.cc:407] Connection header received
0619 16:45:48.966786 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:48.966790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:48.966854 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:48.966955 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:48.967289 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:48.968167 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:48.969419 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:48.969791 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:48.969923 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:48.970996 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:48.971000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:48.971003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:48.971020 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:48.971055 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:48.971071 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:48.971186 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:48.971374 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 16:45:48.971431 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:45:48.972061 21802 raft_consensus.cc:504] T 00000000000000000000000000000000 P de87b2f9fbf948338905101510a6a269 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de87b2f9fbf948338905101510a6a269" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43937 } }
I20260619 16:45:48.972506 21802 leader_election.cc:302] T 00000000000000000000000000000000 P de87b2f9fbf948338905101510a6a269 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de87b2f9fbf948338905101510a6a269; no voters:
I20260619 16:45:48.972906 21802 leader_election.cc:288] T 00000000000000000000000000000000 P de87b2f9fbf948338905101510a6a269 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:45:48.973052 21812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de87b2f9fbf948338905101510a6a269 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:48.973284 21812 raft_consensus.cc:686] T 00000000000000000000000000000000 P de87b2f9fbf948338905101510a6a269 [term 1 LEADER]: Becoming Leader. State: Replica: de87b2f9fbf948338905101510a6a269, State: Running, Role: LEADER
I20260619 16:45:48.973627 21812 consensus_queue.cc:237] T 00000000000000000000000000000000 P de87b2f9fbf948338905101510a6a269 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de87b2f9fbf948338905101510a6a269" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43937 } }
I20260619 16:45:48.974361 21802 sys_catalog.cc:564] T 00000000000000000000000000000000 P de87b2f9fbf948338905101510a6a269 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:45:48.975085 21812 sys_catalog.cc:455] T 00000000000000000000000000000000 P de87b2f9fbf948338905101510a6a269 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de87b2f9fbf948338905101510a6a269. Latest consensus state: current_term: 1 leader_uuid: "de87b2f9fbf948338905101510a6a269" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de87b2f9fbf948338905101510a6a269" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43937 } } }
I20260619 16:45:48.975477 21812 sys_catalog.cc:458] T 00000000000000000000000000000000 P de87b2f9fbf948338905101510a6a269 [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:48.975613 21814 sys_catalog.cc:455] T 00000000000000000000000000000000 P de87b2f9fbf948338905101510a6a269 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de87b2f9fbf948338905101510a6a269" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de87b2f9fbf948338905101510a6a269" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43937 } } }
I20260619 16:45:48.975893 21814 sys_catalog.cc:458] T 00000000000000000000000000000000 P de87b2f9fbf948338905101510a6a269 [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:48.977072 21821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:45:48.980686 21821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:45:48.987350 21821 catalog_manager.cc:1347] Generated new cluster ID: a3e13800e842400498b42b0c0a6bfd98
I20260619 16:45:48.987454 21821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:45:49.009799 21821 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:45:49.011516 21821 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:45:49.018303 21821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de87b2f9fbf948338905101510a6a269: Generated new TSK 0
I20260619 16:45:49.019142 21821 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:45:49.028049 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 21734
W20260619 16:45:49.033864 21719 connection.cc:535] client connection to 127.0.0.1:43937 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43937): Connection reset by peer (error 104)
2026-06-19T16:45:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1521 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:45:49 dist-test-slave-03cd krb5kdc[21833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:45:49 dist-test-slave-03cd krb5kdc[21833](info): set up 2 sockets
Jun 19 16:45:49 dist-test-slave-03cd krb5kdc[21833](info): commencing operation
krb5kdc: starting...
W20260619 16:45:50.088449 392 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:45:50 dist-test-slave-03cd krb5kdc[21833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:45:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:45:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:45:50.224134 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44531
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:50.353173 21846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:50.353506 21846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:50.353623 21846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:50.357825 21846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:45:50.357955 21846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:50.357995 21846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:50.358033 21846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:45:50.358065 21846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:45:50.364123 21846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44531
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887259574980-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:50.365425 21846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:50.366554 21846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:50.376634 21851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:50.376910 21846 server_base.cc:1034] running on GCE node
W20260619 16:45:50.376646 21854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:50.376646 21852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:50.377701 21846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:50.378957 21846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:50.380158 21846 hybrid_clock.cc:648] HybridClock initialized: now 1781887550380139 us; error 34 us; skew 500 ppm
Jun 19 16:45:50 dist-test-slave-03cd krb5kdc[21833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:50.383242 21846 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:45:50.384440 21846 webserver.cc:460] Webserver started at http://127.0.0.1:45895/ using document root <none> and password file <none>
I20260619 16:45:50.385069 21846 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:50.385174 21846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:50.385447 21846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:50.387892 21846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "389250365d244bc4a63f875415ad9b1a"
format_stamp: "Formatted at 2026-06-19 16:45:50 on dist-test-slave-03cd"
server_key: "d13f5ca8109dc188d564d395c6a28479"
server_key_iv: "b95e9d5efbe394c1eb9ab97de44b01bc"
server_key_version: "encryptionkey@0"
I20260619 16:45:50.388557 21846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "389250365d244bc4a63f875415ad9b1a"
format_stamp: "Formatted at 2026-06-19 16:45:50 on dist-test-slave-03cd"
server_key: "d13f5ca8109dc188d564d395c6a28479"
server_key_iv: "b95e9d5efbe394c1eb9ab97de44b01bc"
server_key_version: "encryptionkey@0"
I20260619 16:45:50.393206 21846 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:45:50.396176 21861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:50.397596 21846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:45:50.397795 21846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "389250365d244bc4a63f875415ad9b1a"
format_stamp: "Formatted at 2026-06-19 16:45:50 on dist-test-slave-03cd"
server_key: "d13f5ca8109dc188d564d395c6a28479"
server_key_iv: "b95e9d5efbe394c1eb9ab97de44b01bc"
server_key_version: "encryptionkey@0"
I20260619 16:45:50.397946 21846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:50.403029 21846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:50.406505 21846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:50.406739 21846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:50.414633 21846 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44531
I20260619 16:45:50.414655 21913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44531 every 8 connection(s)
I20260619 16:45:50.415680 21846 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:45:50.418613 21914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:50.421365 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 21846
I20260619 16:45:50.421573 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:45:50.421854 392 external_mini_cluster.cc:1428] Setting key fb1576823ab7eba2ff4ef9bfec88ae53
I20260619 16:45:50.424844 21914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 389250365d244bc4a63f875415ad9b1a: Bootstrap starting.
I20260619 16:45:50.427489 21914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 389250365d244bc4a63f875415ad9b1a: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:50.427752 21917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:50.423460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:37461 (local address 127.0.0.1:44531)
0619 16:45:50.423973 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:50.423998 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:50.424035 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:45:50.424694 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:50.424716 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:50.425063 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:50.425350 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:50.426277 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:50.426282 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:50.426294 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:50.426305 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:50.426361 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:50.426437 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:50.426555 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:50.426766 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 16:45:50.426948 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":224,"threads_started":1}
I20260619 16:45:50.428385 21914 log.cc:824] T 00000000000000000000000000000000 P 389250365d244bc4a63f875415ad9b1a: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:50.430611 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:45:50.430843 21914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 389250365d244bc4a63f875415ad9b1a: No bootstrap required, opened a new log
I20260619 16:45:50.434427 21917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:50.432671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:35291 (local address 127.0.0.1:44531)
0619 16:45:50.432809 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:50.432814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:50.432938 (+ 124us) server_negotiation.cc:407] Connection header received
0619 16:45:50.433032 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:50.433035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:50.433074 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:50.433161 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:50.433917 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:50.433921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:50.433923 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:50.433935 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:50.433966 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:50.433982 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:50.434106 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:50.434205 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 16:45:50.434257 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:45:50.435494 21914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 389250365d244bc4a63f875415ad9b1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389250365d244bc4a63f875415ad9b1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44531 } }
I20260619 16:45:50.435830 21914 raft_consensus.cc:374] T 00000000000000000000000000000000 P 389250365d244bc4a63f875415ad9b1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:50.435884 21914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 389250365d244bc4a63f875415ad9b1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 389250365d244bc4a63f875415ad9b1a, State: Initialized, Role: FOLLOWER
I20260619 16:45:50.436318 21914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 389250365d244bc4a63f875415ad9b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389250365d244bc4a63f875415ad9b1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44531 } }
I20260619 16:45:50.436445 21914 raft_consensus.cc:388] T 00000000000000000000000000000000 P 389250365d244bc4a63f875415ad9b1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:45:50.436502 21914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 389250365d244bc4a63f875415ad9b1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:45:50.436586 21914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 389250365d244bc4a63f875415ad9b1a [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:50.437721 21914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 389250365d244bc4a63f875415ad9b1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389250365d244bc4a63f875415ad9b1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44531 } }
I20260619 16:45:50.438047 21914 leader_election.cc:302] T 00000000000000000000000000000000 P 389250365d244bc4a63f875415ad9b1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 389250365d244bc4a63f875415ad9b1a; no voters:
I20260619 16:45:50.438364 21914 leader_election.cc:288] T 00000000000000000000000000000000 P 389250365d244bc4a63f875415ad9b1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:45:50.438542 21924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 389250365d244bc4a63f875415ad9b1a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:50.438803 21924 raft_consensus.cc:686] T 00000000000000000000000000000000 P 389250365d244bc4a63f875415ad9b1a [term 1 LEADER]: Becoming Leader. State: Replica: 389250365d244bc4a63f875415ad9b1a, State: Running, Role: LEADER
I20260619 16:45:50.439231 21924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 389250365d244bc4a63f875415ad9b1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389250365d244bc4a63f875415ad9b1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44531 } }
I20260619 16:45:50.439741 21914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 389250365d244bc4a63f875415ad9b1a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:45:50.441188 21925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 389250365d244bc4a63f875415ad9b1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "389250365d244bc4a63f875415ad9b1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389250365d244bc4a63f875415ad9b1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44531 } } }
I20260619 16:45:50.441236 21926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 389250365d244bc4a63f875415ad9b1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 389250365d244bc4a63f875415ad9b1a. Latest consensus state: current_term: 1 leader_uuid: "389250365d244bc4a63f875415ad9b1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389250365d244bc4a63f875415ad9b1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44531 } } }
I20260619 16:45:50.441344 21925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 389250365d244bc4a63f875415ad9b1a [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:50.441350 21926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 389250365d244bc4a63f875415ad9b1a [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:50.441813 21936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:45:50.445103 21936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:45:50.451049 21936 catalog_manager.cc:1347] Generated new cluster ID: 80b87924776b4c2f95f5ace5ed889de9
I20260619 16:45:50.451186 21936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:45:50.483786 21936 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:45:50.484220 21936 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:45:50.495370 21936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 389250365d244bc4a63f875415ad9b1a: Generated new TSK 0
I20260619 16:45:50.496392 21936 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:45:50.499779 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 21846
2026-06-19T16:45:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1471 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:45:50 dist-test-slave-03cd krb5kdc[21945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:45:50 dist-test-slave-03cd krb5kdc[21945](info): set up 2 sockets
Jun 19 16:45:50 dist-test-slave-03cd krb5kdc[21945](info): commencing operation
krb5kdc: starting...
W20260619 16:45:51.555939 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:45:51 dist-test-slave-03cd krb5kdc[21945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:45:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:45:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:45:51.685328 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34087
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:51.799219 21958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:51.799613 21958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:51.799734 21958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:51.804242 21958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:45:51.804383 21958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:51.804425 21958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:51.804464 21958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:45:51.804497 21958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:45:51.810346 21958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34087
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:51.811546 21958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:51.812670 21958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:51.821803 21966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:51.821794 21964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:51.821794 21963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:51.822233 21958 server_base.cc:1034] running on GCE node
I20260619 16:45:51.823104 21958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:51.824586 21958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:51.825793 21958 hybrid_clock.cc:648] HybridClock initialized: now 1781887551825764 us; error 49 us; skew 500 ppm
Jun 19 16:45:51 dist-test-slave-03cd krb5kdc[21945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:51.829268 21958 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:45:51.830368 21958 webserver.cc:460] Webserver started at http://127.0.98.62:35913/ using document root <none> and password file <none>
I20260619 16:45:51.830989 21958 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:51.831089 21958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:51.831338 21958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:51.833639 21958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "bd105c09b946480189cb037e61c23a5e"
format_stamp: "Formatted at 2026-06-19 16:45:51 on dist-test-slave-03cd"
server_key: "dd5005ed9760d7ad0a7220d3d645bb62"
server_key_iv: "b9e61fffbd87ce606274da2a9d85408d"
server_key_version: "encryptionkey@0"
I20260619 16:45:51.834245 21958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "bd105c09b946480189cb037e61c23a5e"
format_stamp: "Formatted at 2026-06-19 16:45:51 on dist-test-slave-03cd"
server_key: "dd5005ed9760d7ad0a7220d3d645bb62"
server_key_iv: "b9e61fffbd87ce606274da2a9d85408d"
server_key_version: "encryptionkey@0"
I20260619 16:45:51.838454 21958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:45:51.841457 21973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:51.842983 21958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:45:51.843189 21958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "bd105c09b946480189cb037e61c23a5e"
format_stamp: "Formatted at 2026-06-19 16:45:51 on dist-test-slave-03cd"
server_key: "dd5005ed9760d7ad0a7220d3d645bb62"
server_key_iv: "b9e61fffbd87ce606274da2a9d85408d"
server_key_version: "encryptionkey@0"
I20260619 16:45:51.843348 21958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:51.881914 21958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:51.886271 21958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:51.886539 21958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:51.894968 21958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34087
I20260619 16:45:51.894986 22025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34087 every 8 connection(s)
I20260619 16:45:51.896073 21958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:45:51.899246 22026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:51.903144 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 21958
I20260619 16:45:51.903301 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:45:51.903614 392 external_mini_cluster.cc:1428] Setting key f77a2fc7bd4afd8720580af9fc6f9148
I20260619 16:45:51.906508 22026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd105c09b946480189cb037e61c23a5e: Bootstrap starting.
I20260619 16:45:51.909148 22026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd105c09b946480189cb037e61c23a5e: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:51.910015 22026 log.cc:824] T 00000000000000000000000000000000 P bd105c09b946480189cb037e61c23a5e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:51.912405 22026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd105c09b946480189cb037e61c23a5e: No bootstrap required, opened a new log
Jun 19 16:45:51 dist-test-slave-03cd krb5kdc[21945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:45:51.916569 22026 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd105c09b946480189cb037e61c23a5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd105c09b946480189cb037e61c23a5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34087 } }
I20260619 16:45:51.916863 22026 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd105c09b946480189cb037e61c23a5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:51.916944 22026 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd105c09b946480189cb037e61c23a5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd105c09b946480189cb037e61c23a5e, State: Initialized, Role: FOLLOWER
I20260619 16:45:51.917608 22026 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd105c09b946480189cb037e61c23a5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd105c09b946480189cb037e61c23a5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34087 } }
I20260619 16:45:51.917801 22026 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd105c09b946480189cb037e61c23a5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:45:51.917908 22026 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd105c09b946480189cb037e61c23a5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:45:51.918056 22026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd105c09b946480189cb037e61c23a5e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:51.919128 22026 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd105c09b946480189cb037e61c23a5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd105c09b946480189cb037e61c23a5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34087 } }
I20260619 16:45:51.919550 22026 leader_election.cc:302] T 00000000000000000000000000000000 P bd105c09b946480189cb037e61c23a5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd105c09b946480189cb037e61c23a5e; no voters:
I20260619 16:45:51.919996 22026 leader_election.cc:288] T 00000000000000000000000000000000 P bd105c09b946480189cb037e61c23a5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:45:51.920289 22031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd105c09b946480189cb037e61c23a5e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:51.920550 22031 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd105c09b946480189cb037e61c23a5e [term 1 LEADER]: Becoming Leader. State: Replica: bd105c09b946480189cb037e61c23a5e, State: Running, Role: LEADER
I20260619 16:45:51.920984 22031 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd105c09b946480189cb037e61c23a5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd105c09b946480189cb037e61c23a5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34087 } }
I20260619 16:45:51.921343 22029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:51.905336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39473 (local address 127.0.98.62:34087)
0619 16:45:51.905866 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:51.905882 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:51.905933 (+ 51us) server_negotiation.cc:407] Connection header received
0619 16:45:51.906756 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:51.906777 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:51.907158 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:51.907601 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:51.908454 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:51.909541 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:51.910505 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:51.910999 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:51.911177 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:51.914321 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:51.914327 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:51.914344 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:51.914378 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:51.917253 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:51.918155 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:51.918165 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:51.918172 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:51.918322 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:51.918714 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:51.918717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:51.918719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:51.919195 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:51.919459 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:51.919867 (+ 408us) server_negotiation.cc:299] Negotiation successful
0619 16:45:51.920178 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":174,"threads_started":1}
I20260619 16:45:51.921514 22026 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd105c09b946480189cb037e61c23a5e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:45:51.924746 22032 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd105c09b946480189cb037e61c23a5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd105c09b946480189cb037e61c23a5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd105c09b946480189cb037e61c23a5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34087 } } }
I20260619 16:45:51.925016 22032 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd105c09b946480189cb037e61c23a5e [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:51.925038 22033 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd105c09b946480189cb037e61c23a5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd105c09b946480189cb037e61c23a5e. Latest consensus state: current_term: 1 leader_uuid: "bd105c09b946480189cb037e61c23a5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd105c09b946480189cb037e61c23a5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34087 } } }
I20260619 16:45:51.925200 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:45:51.925204 22033 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd105c09b946480189cb037e61c23a5e [sys.catalog]: This master's current role is: LEADER
W20260619 16:45:51.927645 22045 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bd105c09b946480189cb037e61c23a5e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:45:51.927883 22045 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:45:51.928001 22052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:45:51.929117 22052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:45:51.935863 22052 catalog_manager.cc:1347] Generated new cluster ID: 5a67f89286964dc0a77c9502764ca502
I20260619 16:45:51.935945 22052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:45:51.938906 22029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:51.927935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36209 (local address 127.0.98.62:34087)
0619 16:45:51.928115 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:51.928120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:51.928632 (+ 512us) server_negotiation.cc:407] Connection header received
0619 16:45:51.928746 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:51.928749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:51.928803 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:51.928904 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:51.929776 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:51.930669 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:51.931998 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:51.932373 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:51.932516 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:51.934895 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:51.934899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:51.934902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:51.934935 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:51.936797 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:51.937402 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:51.937406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:51.937409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:51.937508 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:51.937896 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:51.937899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:51.937901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:51.938099 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:51.938286 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:51.938612 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 16:45:51.938724 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:45:51.949277 22052 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:45:51.950526 22052 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:45:51.959549 22052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd105c09b946480189cb037e61c23a5e: Generated new TSK 0
I20260619 16:45:51.960644 22052 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:45:51.968266 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 21958
W20260619 16:45:51.974687 21943 connection.cc:535] client connection to 127.0.98.62:34087 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:34087): Connection reset by peer (error 104)
2026-06-19T16:45:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1469 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:45:52 dist-test-slave-03cd krb5kdc[22057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:45:52 dist-test-slave-03cd krb5kdc[22057](info): set up 2 sockets
Jun 19 16:45:52 dist-test-slave-03cd krb5kdc[22057](info): commencing operation
krb5kdc: starting...
W20260619 16:45:53.048144 392 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:45:53 dist-test-slave-03cd krb5kdc[22057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:45:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:45:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:45:53.180045 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38065
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:53.297521 22070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:53.297871 22070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:53.297963 22070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:53.303864 22070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:45:53.303996 22070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:53.304036 22070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:53.304075 22070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:45:53.304109 22070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:45:53.310127 22070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38065
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:53.311371 22070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:53.312485 22070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:53.322384 22075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:53.322422 22076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:53.322422 22078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:53.322831 22070 server_base.cc:1034] running on GCE node
I20260619 16:45:53.323515 22070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:53.324843 22070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:53.326082 22070 hybrid_clock.cc:648] HybridClock initialized: now 1781887553326053 us; error 52 us; skew 500 ppm
Jun 19 16:45:53 dist-test-slave-03cd krb5kdc[22057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:53.329583 22070 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:45:53.330700 22070 webserver.cc:460] Webserver started at http://127.0.98.62:33345/ using document root <none> and password file <none>
I20260619 16:45:53.331360 22070 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:53.331470 22070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:53.331805 22070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:53.334154 22070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "5b98c30261f940669df69f0b991849df"
format_stamp: "Formatted at 2026-06-19 16:45:53 on dist-test-slave-03cd"
server_key: "83d990f712a89950e20bd460b188bb18"
server_key_iv: "7098fba60a3df0a148be6543a1d1f368"
server_key_version: "encryptionkey@0"
I20260619 16:45:53.334831 22070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "5b98c30261f940669df69f0b991849df"
format_stamp: "Formatted at 2026-06-19 16:45:53 on dist-test-slave-03cd"
server_key: "83d990f712a89950e20bd460b188bb18"
server_key_iv: "7098fba60a3df0a148be6543a1d1f368"
server_key_version: "encryptionkey@0"
I20260619 16:45:53.339273 22070 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:45:53.342000 22085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:53.343340 22070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:45:53.343544 22070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "5b98c30261f940669df69f0b991849df"
format_stamp: "Formatted at 2026-06-19 16:45:53 on dist-test-slave-03cd"
server_key: "83d990f712a89950e20bd460b188bb18"
server_key_iv: "7098fba60a3df0a148be6543a1d1f368"
server_key_version: "encryptionkey@0"
I20260619 16:45:53.343698 22070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:53.357663 22070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:53.361435 22070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:53.361693 22070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:53.369652 22070 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38065
I20260619 16:45:53.369668 22137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38065 every 8 connection(s)
I20260619 16:45:53.370734 22070 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:45:53.373746 22138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:53.377589 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 22070
I20260619 16:45:53.377709 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:45:53.378022 392 external_mini_cluster.cc:1428] Setting key a9f3badd3882b37ac821fe4a9ba29132
I20260619 16:45:53.380599 22138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b98c30261f940669df69f0b991849df: Bootstrap starting.
I20260619 16:45:53.383152 22138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b98c30261f940669df69f0b991849df: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:53.384059 22138 log.cc:824] T 00000000000000000000000000000000 P 5b98c30261f940669df69f0b991849df: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:53.386108 22138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b98c30261f940669df69f0b991849df: No bootstrap required, opened a new log
I20260619 16:45:53.388032 22141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:53.379707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57445 (local address 127.0.98.62:38065)
0619 16:45:53.380195 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:53.380216 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:53.380253 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:45:53.380977 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:53.380995 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:53.381355 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:53.381642 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:53.382086 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:53.383172 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:53.384430 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:53.384971 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:53.385158 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:53.386110 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:53.386115 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:53.386129 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:53.386606 (+ 477us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:53.386664 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:53.386688 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:53.386833 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:53.387008 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 16:45:53.387158 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":158,"threads_started":1}
I20260619 16:45:53.391256 22138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b98c30261f940669df69f0b991849df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b98c30261f940669df69f0b991849df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38065 } }
I20260619 16:45:53.391590 22138 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b98c30261f940669df69f0b991849df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:53.391696 22138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b98c30261f940669df69f0b991849df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b98c30261f940669df69f0b991849df, State: Initialized, Role: FOLLOWER
I20260619 16:45:53.391988 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:45:53.392232 22138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b98c30261f940669df69f0b991849df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b98c30261f940669df69f0b991849df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38065 } }
I20260619 16:45:53.392400 22138 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b98c30261f940669df69f0b991849df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:45:53.392488 22138 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b98c30261f940669df69f0b991849df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:45:53.392614 22138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b98c30261f940669df69f0b991849df [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:53.393760 22138 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b98c30261f940669df69f0b991849df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b98c30261f940669df69f0b991849df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38065 } }
I20260619 16:45:53.394166 22138 leader_election.cc:302] T 00000000000000000000000000000000 P 5b98c30261f940669df69f0b991849df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b98c30261f940669df69f0b991849df; no voters:
I20260619 16:45:53.394555 22138 leader_election.cc:288] T 00000000000000000000000000000000 P 5b98c30261f940669df69f0b991849df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:45:53.394622 22148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b98c30261f940669df69f0b991849df [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:53.394886 22148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b98c30261f940669df69f0b991849df [term 1 LEADER]: Becoming Leader. State: Replica: 5b98c30261f940669df69f0b991849df, State: Running, Role: LEADER
I20260619 16:45:53.395273 22148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b98c30261f940669df69f0b991849df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b98c30261f940669df69f0b991849df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38065 } }
I20260619 16:45:53.396193 22138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b98c30261f940669df69f0b991849df [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:45:53.397444 22150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b98c30261f940669df69f0b991849df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b98c30261f940669df69f0b991849df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b98c30261f940669df69f0b991849df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38065 } } }
I20260619 16:45:53.397444 22149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b98c30261f940669df69f0b991849df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b98c30261f940669df69f0b991849df. Latest consensus state: current_term: 1 leader_uuid: "5b98c30261f940669df69f0b991849df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b98c30261f940669df69f0b991849df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38065 } } }
I20260619 16:45:53.397620 22150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b98c30261f940669df69f0b991849df [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:53.397620 22149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b98c30261f940669df69f0b991849df [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:53.399732 22141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:53.394260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38453 (local address 127.0.98.62:38065)
0619 16:45:53.394422 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:53.394428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:53.394662 (+ 234us) server_negotiation.cc:407] Connection header received
0619 16:45:53.394871 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:53.394876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:53.394938 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:53.395037 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:53.395445 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:53.396226 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:53.397495 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:53.397876 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:53.397988 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:53.399168 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:53.399172 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:53.399175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:53.399190 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:53.399224 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:53.399235 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:53.399339 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:53.399492 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 16:45:53.399538 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260619 16:45:53.401361 22163 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5b98c30261f940669df69f0b991849df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:45:53.401436 22163 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:45:53.401526 22158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:45:53.402249 22158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:45:53.408224 22158 catalog_manager.cc:1347] Generated new cluster ID: c551fc65b1a64c70b942d30e03382540
I20260619 16:45:53.408320 22158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:45:53.420523 22158 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:45:53.421519 22158 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:45:53.429765 22158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b98c30261f940669df69f0b991849df: Generated new TSK 0
I20260619 16:45:53.430541 22158 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:45:53.440469 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 22070
W20260619 16:45:53.445848 22055 connection.cc:535] client connection to 127.0.98.62:38065 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:38065): Connection reset by peer (error 104)
2026-06-19T16:45:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1470 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:45:53 dist-test-slave-03cd krb5kdc[22169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:45:53 dist-test-slave-03cd krb5kdc[22169](info): set up 2 sockets
Jun 19 16:45:53 dist-test-slave-03cd krb5kdc[22169](info): commencing operation
krb5kdc: starting...
W20260619 16:45:54.516024 392 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:45:54 dist-test-slave-03cd krb5kdc[22169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:45:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:45:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:45:54.648083 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38725
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:54.762310 22182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:54.762616 22182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:54.762660 22182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:54.766974 22182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:45:54.767051 22182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:54.767071 22182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:54.767088 22182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:45:54.767103 22182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:45:54.772852 22182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38725
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:54.774086 22182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:54.775137 22182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:54.784061 22188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:54.784067 22190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:54.784106 22187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:54.784142 22182 server_base.cc:1034] running on GCE node
I20260619 16:45:54.784927 22182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:54.786206 22182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:54.787436 22182 hybrid_clock.cc:648] HybridClock initialized: now 1781887554787402 us; error 50 us; skew 500 ppm
Jun 19 16:45:54 dist-test-slave-03cd krb5kdc[22169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:54.796558 22182 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:45:54.797793 22182 webserver.cc:460] Webserver started at http://127.0.98.62:39555/ using document root <none> and password file <none>
I20260619 16:45:54.798472 22182 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:54.798579 22182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:54.798856 22182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:54.801159 22182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "aed73ce782ef4997ac865ed7765c67fd"
format_stamp: "Formatted at 2026-06-19 16:45:54 on dist-test-slave-03cd"
server_key: "967b8dda2fb033a0b5942da9272b89ea"
server_key_iv: "c002adefe824e1c8487f585562e5d163"
server_key_version: "encryptionkey@0"
I20260619 16:45:54.801780 22182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "aed73ce782ef4997ac865ed7765c67fd"
format_stamp: "Formatted at 2026-06-19 16:45:54 on dist-test-slave-03cd"
server_key: "967b8dda2fb033a0b5942da9272b89ea"
server_key_iv: "c002adefe824e1c8487f585562e5d163"
server_key_version: "encryptionkey@0"
I20260619 16:45:54.805902 22182 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260619 16:45:54.808657 22197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:54.810076 22182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:45:54.810256 22182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "aed73ce782ef4997ac865ed7765c67fd"
format_stamp: "Formatted at 2026-06-19 16:45:54 on dist-test-slave-03cd"
server_key: "967b8dda2fb033a0b5942da9272b89ea"
server_key_iv: "c002adefe824e1c8487f585562e5d163"
server_key_version: "encryptionkey@0"
I20260619 16:45:54.810400 22182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:54.815840 22182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:54.818989 22182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:54.819216 22182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:54.827800 22182 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38725
I20260619 16:45:54.827797 22249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38725 every 8 connection(s)
I20260619 16:45:54.828814 22182 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:45:54.831775 22250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:54.835147 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 22182
I20260619 16:45:54.835266 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:45:54.835572 392 external_mini_cluster.cc:1428] Setting key bc51a7f0059a198a9fbe07830d01a3c0
I20260619 16:45:54.838004 22250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aed73ce782ef4997ac865ed7765c67fd: Bootstrap starting.
I20260619 16:45:54.840515 22250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aed73ce782ef4997ac865ed7765c67fd: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:54.841257 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:54.837166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58671 (local address 127.0.98.62:38725)
0619 16:45:54.837700 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:54.837715 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:54.837754 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:45:54.838496 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:54.838515 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:54.838811 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:54.839176 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:54.840024 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:54.840029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:54.840040 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:54.840051 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:54.840106 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:54.840126 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:54.840247 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:54.840345 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 16:45:54.840511 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":157,"threads_started":1}
I20260619 16:45:54.841344 22250 log.cc:824] T 00000000000000000000000000000000 P aed73ce782ef4997ac865ed7765c67fd: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:54.843433 22250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aed73ce782ef4997ac865ed7765c67fd: No bootstrap required, opened a new log
I20260619 16:45:54.844230 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:45:54.847476 22250 raft_consensus.cc:348] T 00000000000000000000000000000000 P aed73ce782ef4997ac865ed7765c67fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed73ce782ef4997ac865ed7765c67fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38725 } }
I20260619 16:45:54.847796 22250 raft_consensus.cc:374] T 00000000000000000000000000000000 P aed73ce782ef4997ac865ed7765c67fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:54.847898 22250 raft_consensus.cc:729] T 00000000000000000000000000000000 P aed73ce782ef4997ac865ed7765c67fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aed73ce782ef4997ac865ed7765c67fd, State: Initialized, Role: FOLLOWER
I20260619 16:45:54.847945 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:54.846298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51313 (local address 127.0.98.62:38725)
0619 16:45:54.846524 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:54.846530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:54.846545 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:45:54.846670 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:54.846673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:54.846707 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:54.846798 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:54.847513 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:54.847516 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:54.847519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:54.847530 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:54.847556 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:54.847575 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:54.847664 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:54.847761 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 16:45:54.847800 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260619 16:45:54.848407 22250 consensus_queue.cc:260] T 00000000000000000000000000000000 P aed73ce782ef4997ac865ed7765c67fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed73ce782ef4997ac865ed7765c67fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38725 } }
I20260619 16:45:54.848563 22250 raft_consensus.cc:388] T 00000000000000000000000000000000 P aed73ce782ef4997ac865ed7765c67fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:45:54.848650 22250 raft_consensus.cc:482] T 00000000000000000000000000000000 P aed73ce782ef4997ac865ed7765c67fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:45:54.848752 22250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aed73ce782ef4997ac865ed7765c67fd [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:54.849761 22250 raft_consensus.cc:504] T 00000000000000000000000000000000 P aed73ce782ef4997ac865ed7765c67fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed73ce782ef4997ac865ed7765c67fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38725 } }
I20260619 16:45:54.850106 22250 leader_election.cc:302] T 00000000000000000000000000000000 P aed73ce782ef4997ac865ed7765c67fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aed73ce782ef4997ac865ed7765c67fd; no voters:
I20260619 16:45:54.850410 22250 leader_election.cc:288] T 00000000000000000000000000000000 P aed73ce782ef4997ac865ed7765c67fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:45:54.850651 22260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aed73ce782ef4997ac865ed7765c67fd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:54.850878 22260 raft_consensus.cc:686] T 00000000000000000000000000000000 P aed73ce782ef4997ac865ed7765c67fd [term 1 LEADER]: Becoming Leader. State: Replica: aed73ce782ef4997ac865ed7765c67fd, State: Running, Role: LEADER
I20260619 16:45:54.851366 22260 consensus_queue.cc:237] T 00000000000000000000000000000000 P aed73ce782ef4997ac865ed7765c67fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed73ce782ef4997ac865ed7765c67fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38725 } }
I20260619 16:45:54.851914 22250 sys_catalog.cc:564] T 00000000000000000000000000000000 P aed73ce782ef4997ac865ed7765c67fd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:45:54.853717 22261 sys_catalog.cc:455] T 00000000000000000000000000000000 P aed73ce782ef4997ac865ed7765c67fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aed73ce782ef4997ac865ed7765c67fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed73ce782ef4997ac865ed7765c67fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38725 } } }
I20260619 16:45:54.853742 22262 sys_catalog.cc:455] T 00000000000000000000000000000000 P aed73ce782ef4997ac865ed7765c67fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader aed73ce782ef4997ac865ed7765c67fd. Latest consensus state: current_term: 1 leader_uuid: "aed73ce782ef4997ac865ed7765c67fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed73ce782ef4997ac865ed7765c67fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38725 } } }
I20260619 16:45:54.853870 22261 sys_catalog.cc:458] T 00000000000000000000000000000000 P aed73ce782ef4997ac865ed7765c67fd [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:54.853900 22262 sys_catalog.cc:458] T 00000000000000000000000000000000 P aed73ce782ef4997ac865ed7765c67fd [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:54.854293 22269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:45:54.857472 22269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:45:54.863816 22269 catalog_manager.cc:1347] Generated new cluster ID: ed0b250601a349ca89385b4c7544495c
I20260619 16:45:54.863910 22269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:45:54.883320 22269 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:45:54.883720 22269 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:45:54.897846 22269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aed73ce782ef4997ac865ed7765c67fd: Generated new TSK 0
I20260619 16:45:54.898779 22269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:45:54.908804 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 22182
2026-06-19T16:45:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1468 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:45:54 dist-test-slave-03cd krb5kdc[22281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:45:54 dist-test-slave-03cd krb5kdc[22281](info): set up 2 sockets
Jun 19 16:45:54 dist-test-slave-03cd krb5kdc[22281](info): commencing operation
krb5kdc: starting...
W20260619 16:45:55.980047 392 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:45:56 dist-test-slave-03cd krb5kdc[22281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:45:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:45:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:45:56.116441 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36101
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:56.232056 22294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:56.232425 22294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:56.232519 22294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:56.236918 22294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:45:56.237047 22294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:56.237108 22294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:56.237170 22294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:45:56.237207 22294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:45:56.243046 22294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:56.244393 22294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:56.245501 22294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:56.254760 22302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:56.254855 22300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:56.254855 22299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:56.255172 22294 server_base.cc:1034] running on GCE node
I20260619 16:45:56.255877 22294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:56.257172 22294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:56.258352 22294 hybrid_clock.cc:648] HybridClock initialized: now 1781887556258334 us; error 35 us; skew 500 ppm
Jun 19 16:45:56 dist-test-slave-03cd krb5kdc[22281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:56.261547 22294 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:45:56.262607 22294 webserver.cc:460] Webserver started at http://127.0.98.62:42705/ using document root <none> and password file <none>
I20260619 16:45:56.263262 22294 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:56.263365 22294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:56.263641 22294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:56.266007 22294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "ef66d5b4039142188584570ea7752f59"
format_stamp: "Formatted at 2026-06-19 16:45:56 on dist-test-slave-03cd"
server_key: "2945e568368bade4c051ccd3cf71e4ca"
server_key_iv: "5b8da21e360e97cbbbcf9b1eea4ab23f"
server_key_version: "encryptionkey@0"
I20260619 16:45:56.266645 22294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "ef66d5b4039142188584570ea7752f59"
format_stamp: "Formatted at 2026-06-19 16:45:56 on dist-test-slave-03cd"
server_key: "2945e568368bade4c051ccd3cf71e4ca"
server_key_iv: "5b8da21e360e97cbbbcf9b1eea4ab23f"
server_key_version: "encryptionkey@0"
I20260619 16:45:56.270792 22294 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 16:45:56.273584 22309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:56.274987 22294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:45:56.275149 22294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "ef66d5b4039142188584570ea7752f59"
format_stamp: "Formatted at 2026-06-19 16:45:56 on dist-test-slave-03cd"
server_key: "2945e568368bade4c051ccd3cf71e4ca"
server_key_iv: "5b8da21e360e97cbbbcf9b1eea4ab23f"
server_key_version: "encryptionkey@0"
I20260619 16:45:56.275283 22294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:56.289198 22294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:56.292757 22294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:56.292976 22294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:56.300222 22294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36101
I20260619 16:45:56.300249 22361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36101 every 8 connection(s)
I20260619 16:45:56.301188 22294 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:45:56.303603 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 22294
I20260619 16:45:56.303790 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:45:56.304080 392 external_mini_cluster.cc:1428] Setting key 036fcf421ca187ceea7be6f9e55bcee0
I20260619 16:45:56.304301 22362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:56.310631 22362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef66d5b4039142188584570ea7752f59: Bootstrap starting.
Jun 19 16:45:56 dist-test-slave-03cd krb5kdc[22281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:45:56.313305 22362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef66d5b4039142188584570ea7752f59: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:56.314291 22362 log.cc:824] T 00000000000000000000000000000000 P ef66d5b4039142188584570ea7752f59: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:56.316984 22362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef66d5b4039142188584570ea7752f59: No bootstrap required, opened a new log
I20260619 16:45:56.319293 22365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:56.305422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53435 (local address 127.0.98.62:36101)
0619 16:45:56.305879 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:56.305893 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:56.305928 (+ 35us) server_negotiation.cc:407] Connection header received
0619 16:45:56.306608 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:56.306625 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:56.306963 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:56.307309 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:56.308152 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:56.309145 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:56.310023 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:56.310510 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:56.310679 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:56.313542 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:56.313550 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:56.313574 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:56.313618 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:56.315710 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:56.316286 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:56.316292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:56.316298 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:56.316445 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:56.316750 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:56.316753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:56.316755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:56.317307 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:56.317513 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:56.317725 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 16:45:56.318038 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":153,"threads_started":1}
I20260619 16:45:56.321166 22362 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef66d5b4039142188584570ea7752f59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef66d5b4039142188584570ea7752f59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36101 } }
I20260619 16:45:56.321485 22362 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef66d5b4039142188584570ea7752f59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:56.321578 22362 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef66d5b4039142188584570ea7752f59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef66d5b4039142188584570ea7752f59, State: Initialized, Role: FOLLOWER
I20260619 16:45:56.322202 22362 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef66d5b4039142188584570ea7752f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef66d5b4039142188584570ea7752f59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36101 } }
I20260619 16:45:56.322381 22362 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef66d5b4039142188584570ea7752f59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:45:56.322464 22362 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef66d5b4039142188584570ea7752f59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:45:56.322570 22362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef66d5b4039142188584570ea7752f59 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:56.322906 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:45:56.323726 22362 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef66d5b4039142188584570ea7752f59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef66d5b4039142188584570ea7752f59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36101 } }
I20260619 16:45:56.324106 22362 leader_election.cc:302] T 00000000000000000000000000000000 P ef66d5b4039142188584570ea7752f59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef66d5b4039142188584570ea7752f59; no voters:
I20260619 16:45:56.324482 22362 leader_election.cc:288] T 00000000000000000000000000000000 P ef66d5b4039142188584570ea7752f59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:45:56.324570 22371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef66d5b4039142188584570ea7752f59 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:56.324751 22371 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef66d5b4039142188584570ea7752f59 [term 1 LEADER]: Becoming Leader. State: Replica: ef66d5b4039142188584570ea7752f59, State: Running, Role: LEADER
I20260619 16:45:56.325695 22371 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef66d5b4039142188584570ea7752f59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef66d5b4039142188584570ea7752f59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36101 } }
I20260619 16:45:56.325868 22362 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef66d5b4039142188584570ea7752f59 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:45:56.326889 22371 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef66d5b4039142188584570ea7752f59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef66d5b4039142188584570ea7752f59. Latest consensus state: current_term: 1 leader_uuid: "ef66d5b4039142188584570ea7752f59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef66d5b4039142188584570ea7752f59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36101 } } }
I20260619 16:45:56.327071 22371 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef66d5b4039142188584570ea7752f59 [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:56.327282 22372 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef66d5b4039142188584570ea7752f59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef66d5b4039142188584570ea7752f59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef66d5b4039142188584570ea7752f59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36101 } } }
I20260619 16:45:56.327455 22372 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef66d5b4039142188584570ea7752f59 [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:56.327515 22381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:45:56.330943 22381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:45:56.334848 22365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:56.325108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48793 (local address 127.0.98.62:36101)
0619 16:45:56.325217 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:56.325222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:56.325542 (+ 320us) server_negotiation.cc:407] Connection header received
0619 16:45:56.325692 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:56.325694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:56.325740 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:56.325845 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:56.326651 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:56.327520 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:56.328740 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:56.329076 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:56.329191 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:56.331189 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:56.331193 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:56.331196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:56.331223 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:56.332868 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:56.333434 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:56.333438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:56.333440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:56.333531 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:56.333883 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:56.333886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:56.333888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:56.334144 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:56.334275 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:56.334561 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 16:45:56.334691 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 16:45:56.337505 22381 catalog_manager.cc:1347] Generated new cluster ID: 0af173dc3142404a97e56fb77f25d941
I20260619 16:45:56.337585 22381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:45:56.358100 22381 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:45:56.359491 22381 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:45:56.372206 22381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef66d5b4039142188584570ea7752f59: Generated new TSK 0
I20260619 16:45:56.373176 22381 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:45:56.377769 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 22294
2026-06-19T16:45:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1469 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:45:56 dist-test-slave-03cd krb5kdc[22393](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:45:56 dist-test-slave-03cd krb5kdc[22393](info): set up 2 sockets
Jun 19 16:45:56 dist-test-slave-03cd krb5kdc[22393](info): commencing operation
krb5kdc: starting...
W20260619 16:45:57.432826 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:45:57 dist-test-slave-03cd krb5kdc[22393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:45:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:45:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:45:57.572502 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36071
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:57.707607 22406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:57.708045 22406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:57.708125 22406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:57.712525 22406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:45:57.712612 22406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:57.712632 22406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:57.712651 22406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:45:57.712667 22406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:45:57.718930 22406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:57.720603 22406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:57.721941 22406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:57.732565 22412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:57.732465 22411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:57.732734 22406 server_base.cc:1034] running on GCE node
W20260619 16:45:57.732620 22414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:57.733645 22406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:57.735093 22406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:57.736451 22406 hybrid_clock.cc:648] HybridClock initialized: now 1781887557736341 us; error 161 us; skew 500 ppm
Jun 19 16:45:57 dist-test-slave-03cd krb5kdc[22393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:57.739977 22406 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:45:57.741220 22406 webserver.cc:460] Webserver started at http://127.0.98.62:41031/ using document root <none> and password file <none>
I20260619 16:45:57.741954 22406 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:57.742069 22406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:57.742352 22406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:57.745137 22406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "1fb4b7379e804c4c90b6311a644d870a"
format_stamp: "Formatted at 2026-06-19 16:45:57 on dist-test-slave-03cd"
server_key: "fd4aa9acf6a5eab2f30965484c742f7e"
server_key_iv: "963507ae197b9fdb6edd80fc25f745a9"
server_key_version: "encryptionkey@0"
I20260619 16:45:57.745906 22406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "1fb4b7379e804c4c90b6311a644d870a"
format_stamp: "Formatted at 2026-06-19 16:45:57 on dist-test-slave-03cd"
server_key: "fd4aa9acf6a5eab2f30965484c742f7e"
server_key_iv: "963507ae197b9fdb6edd80fc25f745a9"
server_key_version: "encryptionkey@0"
I20260619 16:45:57.751580 22406 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 16:45:57.755280 22421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:57.757395 22406 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:45:57.757643 22406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "1fb4b7379e804c4c90b6311a644d870a"
format_stamp: "Formatted at 2026-06-19 16:45:57 on dist-test-slave-03cd"
server_key: "fd4aa9acf6a5eab2f30965484c742f7e"
server_key_iv: "963507ae197b9fdb6edd80fc25f745a9"
server_key_version: "encryptionkey@0"
I20260619 16:45:57.757850 22406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:57.789983 22406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:57.794706 22406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:57.795003 22406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:57.804908 22406 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36071
I20260619 16:45:57.805075 22473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36071 every 8 connection(s)
I20260619 16:45:57.806456 22406 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:45:57.809818 22474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:57.811910 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 22406
I20260619 16:45:57.812091 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:45:57.812400 392 external_mini_cluster.cc:1428] Setting key d7608386dc8fc098d9234f62665e0554
I20260619 16:45:57.817116 22474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fb4b7379e804c4c90b6311a644d870a: Bootstrap starting.
I20260619 16:45:57.819645 22474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fb4b7379e804c4c90b6311a644d870a: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:57.820562 22474 log.cc:824] T 00000000000000000000000000000000 P 1fb4b7379e804c4c90b6311a644d870a: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:57.823589 22474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fb4b7379e804c4c90b6311a644d870a: No bootstrap required, opened a new log
I20260619 16:45:57.824517 22477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:57.814284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34805 (local address 127.0.98.62:36071)
0619 16:45:57.814874 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:57.814895 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:57.814946 (+ 51us) server_negotiation.cc:407] Connection header received
0619 16:45:57.816206 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:57.816234 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:57.816736 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:57.817205 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:57.817732 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:57.818857 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:57.819977 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:57.820578 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:57.820770 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:57.821886 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:57.821893 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:57.821914 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:57.822572 (+ 658us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:57.822655 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:57.822688 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:57.822801 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:57.822977 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 16:45:57.823249 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":182,"threads_started":1}
I20260619 16:45:57.828662 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:45:57.828737 22474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fb4b7379e804c4c90b6311a644d870a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb4b7379e804c4c90b6311a644d870a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36071 } }
I20260619 16:45:57.829088 22474 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fb4b7379e804c4c90b6311a644d870a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:57.829147 22474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fb4b7379e804c4c90b6311a644d870a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fb4b7379e804c4c90b6311a644d870a, State: Initialized, Role: FOLLOWER
I20260619 16:45:57.829747 22474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fb4b7379e804c4c90b6311a644d870a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb4b7379e804c4c90b6311a644d870a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36071 } }
I20260619 16:45:57.829931 22474 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fb4b7379e804c4c90b6311a644d870a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:45:57.830008 22474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fb4b7379e804c4c90b6311a644d870a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:45:57.830119 22474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fb4b7379e804c4c90b6311a644d870a [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:57.831647 22474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fb4b7379e804c4c90b6311a644d870a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb4b7379e804c4c90b6311a644d870a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36071 } }
I20260619 16:45:57.832208 22474 leader_election.cc:302] T 00000000000000000000000000000000 P 1fb4b7379e804c4c90b6311a644d870a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fb4b7379e804c4c90b6311a644d870a; no voters:
I20260619 16:45:57.832671 22474 leader_election.cc:288] T 00000000000000000000000000000000 P 1fb4b7379e804c4c90b6311a644d870a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:45:57.832873 22484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fb4b7379e804c4c90b6311a644d870a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:57.833235 22484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fb4b7379e804c4c90b6311a644d870a [term 1 LEADER]: Becoming Leader. State: Replica: 1fb4b7379e804c4c90b6311a644d870a, State: Running, Role: LEADER
I20260619 16:45:57.833696 22484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fb4b7379e804c4c90b6311a644d870a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb4b7379e804c4c90b6311a644d870a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36071 } }
I20260619 16:45:57.834386 22474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fb4b7379e804c4c90b6311a644d870a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:45:57.835172 22484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fb4b7379e804c4c90b6311a644d870a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fb4b7379e804c4c90b6311a644d870a. Latest consensus state: current_term: 1 leader_uuid: "1fb4b7379e804c4c90b6311a644d870a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb4b7379e804c4c90b6311a644d870a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36071 } } }
I20260619 16:45:57.836134 22484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fb4b7379e804c4c90b6311a644d870a [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:57.835968 22485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fb4b7379e804c4c90b6311a644d870a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fb4b7379e804c4c90b6311a644d870a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb4b7379e804c4c90b6311a644d870a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36071 } } }
I20260619 16:45:57.836536 22485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fb4b7379e804c4c90b6311a644d870a [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:57.837487 22477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:57.831544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48909 (local address 127.0.98.62:36071)
0619 16:45:57.831746 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:57.831752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:57.832042 (+ 290us) server_negotiation.cc:407] Connection header received
0619 16:45:57.832209 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:57.832212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:57.832268 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:57.832348 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:57.832744 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:57.833567 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:57.834790 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:57.835143 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:57.836424 (+ 1281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:57.836575 (+ 151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:57.836581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:57.836586 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:57.836603 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:57.836627 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:57.836640 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:57.837118 (+ 478us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:57.837157 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 16:45:57.837215 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 16:45:57.841571 22499 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1fb4b7379e804c4c90b6311a644d870a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:45:57.841677 22499 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:45:57.841766 22493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:45:57.842690 22493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:45:57.850456 22493 catalog_manager.cc:1347] Generated new cluster ID: 911ad6113414469f830686a05ad04ba8
I20260619 16:45:57.850606 22493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:45:57.873783 22493 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:45:57.875229 22493 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:45:57.882941 22493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fb4b7379e804c4c90b6311a644d870a: Generated new TSK 0
I20260619 16:45:57.884081 22493 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:45:57.897352 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 22406
2026-06-19T16:45:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1521 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:45:57 dist-test-slave-03cd krb5kdc[22505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:45:57 dist-test-slave-03cd krb5kdc[22505](info): set up 2 sockets
Jun 19 16:45:57 dist-test-slave-03cd krb5kdc[22505](info): commencing operation
krb5kdc: starting...
W20260619 16:45:59.008733 392 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:45:59 dist-test-slave-03cd krb5kdc[22505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:45:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:45:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:45:59.151942 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41873
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:59.285210 22518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:59.285552 22518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:59.285643 22518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:59.290015 22518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:45:59.290145 22518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:59.290186 22518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:59.290226 22518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:45:59.290262 22518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:45:59.296185 22518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:45:59.297495 22518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:59.298619 22518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:59.308478 22524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:59.308591 22526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:59.308513 22523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:59.308902 22518 server_base.cc:1034] running on GCE node
I20260619 16:45:59.309805 22518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:59.311445 22518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:59.312808 22518 hybrid_clock.cc:648] HybridClock initialized: now 1781887559312730 us; error 98 us; skew 500 ppm
Jun 19 16:45:59 dist-test-slave-03cd krb5kdc[22505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:59.316983 22518 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:45:59.318449 22518 webserver.cc:460] Webserver started at http://127.0.98.62:39161/ using document root <none> and password file <none>
I20260619 16:45:59.319276 22518 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:59.319375 22518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:59.319711 22518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:45:59.322633 22518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "25c5d61a365942819edc6a5220e46c6f"
format_stamp: "Formatted at 2026-06-19 16:45:59 on dist-test-slave-03cd"
server_key: "080c0fa2ce689601b050aa5806ee2e8d"
server_key_iv: "daa2b811b26b3d3c09e90e6af8d47327"
server_key_version: "encryptionkey@0"
I20260619 16:45:59.323377 22518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "25c5d61a365942819edc6a5220e46c6f"
format_stamp: "Formatted at 2026-06-19 16:45:59 on dist-test-slave-03cd"
server_key: "080c0fa2ce689601b050aa5806ee2e8d"
server_key_iv: "daa2b811b26b3d3c09e90e6af8d47327"
server_key_version: "encryptionkey@0"
I20260619 16:45:59.328634 22518 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 16:45:59.332160 22533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:59.333895 22518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:45:59.334088 22518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "25c5d61a365942819edc6a5220e46c6f"
format_stamp: "Formatted at 2026-06-19 16:45:59 on dist-test-slave-03cd"
server_key: "080c0fa2ce689601b050aa5806ee2e8d"
server_key_iv: "daa2b811b26b3d3c09e90e6af8d47327"
server_key_version: "encryptionkey@0"
I20260619 16:45:59.334254 22518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:59.340160 22518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:59.344182 22518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:59.344434 22518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:59.353444 22518 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41873
I20260619 16:45:59.353513 22585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41873 every 8 connection(s)
I20260619 16:45:59.354737 22518 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:45:59.358098 22586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:59.361263 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 22518
I20260619 16:45:59.361418 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:45:59.361783 392 external_mini_cluster.cc:1428] Setting key 22262588e442bc2b9a7a80722cc404a7
I20260619 16:45:59.365293 22586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25c5d61a365942819edc6a5220e46c6f: Bootstrap starting.
I20260619 16:45:59.368479 22586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25c5d61a365942819edc6a5220e46c6f: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:59.369251 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:59.363856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53067 (local address 127.0.98.62:41873)
0619 16:45:59.364571 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:59.364592 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:59.364645 (+ 53us) server_negotiation.cc:407] Connection header received
0619 16:45:59.365563 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:59.365591 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:59.365994 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:59.366435 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:59.367452 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:59.367458 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:59.367474 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:59.367488 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:59.367561 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:59.367591 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:59.367756 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:59.367922 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 16:45:59.368093 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":203,"threads_started":1}
I20260619 16:45:59.369506 22586 log.cc:824] T 00000000000000000000000000000000 P 25c5d61a365942819edc6a5220e46c6f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:59.372152 22586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25c5d61a365942819edc6a5220e46c6f: No bootstrap required, opened a new log
I20260619 16:45:59.373582 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:45:59.376533 22586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25c5d61a365942819edc6a5220e46c6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c5d61a365942819edc6a5220e46c6f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41873 } }
I20260619 16:45:59.376942 22586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25c5d61a365942819edc6a5220e46c6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:59.377110 22586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25c5d61a365942819edc6a5220e46c6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25c5d61a365942819edc6a5220e46c6f, State: Initialized, Role: FOLLOWER
I20260619 16:45:59.377640 22586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25c5d61a365942819edc6a5220e46c6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c5d61a365942819edc6a5220e46c6f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41873 } }
I20260619 16:45:59.377806 22586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25c5d61a365942819edc6a5220e46c6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:45:59.377902 22586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25c5d61a365942819edc6a5220e46c6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:45:59.378033 22586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25c5d61a365942819edc6a5220e46c6f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:59.378619 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:59.376385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33583 (local address 127.0.98.62:41873)
0619 16:45:59.376543 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:59.376549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:59.376632 (+ 83us) server_negotiation.cc:407] Connection header received
0619 16:45:59.376768 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:59.376771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:59.376817 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:59.376947 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:59.378032 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:59.378045 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:59.378048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:45:59.378065 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:59.378139 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:45:59.378158 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:59.378238 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:59.378433 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 16:45:59.378486 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:45:59.379251 22586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25c5d61a365942819edc6a5220e46c6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c5d61a365942819edc6a5220e46c6f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41873 } }
I20260619 16:45:59.379698 22586 leader_election.cc:302] T 00000000000000000000000000000000 P 25c5d61a365942819edc6a5220e46c6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25c5d61a365942819edc6a5220e46c6f; no voters:
I20260619 16:45:59.380101 22586 leader_election.cc:288] T 00000000000000000000000000000000 P 25c5d61a365942819edc6a5220e46c6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:45:59.380183 22596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25c5d61a365942819edc6a5220e46c6f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:59.380400 22596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25c5d61a365942819edc6a5220e46c6f [term 1 LEADER]: Becoming Leader. State: Replica: 25c5d61a365942819edc6a5220e46c6f, State: Running, Role: LEADER
I20260619 16:45:59.380719 22596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25c5d61a365942819edc6a5220e46c6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c5d61a365942819edc6a5220e46c6f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41873 } }
I20260619 16:45:59.381548 22586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25c5d61a365942819edc6a5220e46c6f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:45:59.382762 22597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25c5d61a365942819edc6a5220e46c6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25c5d61a365942819edc6a5220e46c6f. Latest consensus state: current_term: 1 leader_uuid: "25c5d61a365942819edc6a5220e46c6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c5d61a365942819edc6a5220e46c6f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41873 } } }
I20260619 16:45:59.382716 22598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25c5d61a365942819edc6a5220e46c6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25c5d61a365942819edc6a5220e46c6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c5d61a365942819edc6a5220e46c6f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41873 } } }
I20260619 16:45:59.382972 22597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25c5d61a365942819edc6a5220e46c6f [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:59.383080 22598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25c5d61a365942819edc6a5220e46c6f [sys.catalog]: This master's current role is: LEADER
W20260619 16:45:59.387018 22611 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 25c5d61a365942819edc6a5220e46c6f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:45:59.387138 22611 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:45:59.387245 22605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:45:59.388131 22605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:45:59.396034 22605 catalog_manager.cc:1347] Generated new cluster ID: 266394935fc9464da02b633bdc446d73
I20260619 16:45:59.396152 22605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:45:59.412725 22605 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:45:59.413197 22605 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:45:59.420012 22605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25c5d61a365942819edc6a5220e46c6f: Generated new TSK 0
I20260619 16:45:59.421348 22605 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:45:59.433681 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 22518
2026-06-19T16:45:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1536 ms)
[----------] 18 tests from AuthTokenIssuingTest (26665 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:45:59 dist-test-slave-03cd krb5kdc[22617](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:45:59 dist-test-slave-03cd krb5kdc[22617](info): set up 2 sockets
Jun 19 16:45:59 dist-test-slave-03cd krb5kdc[22617](info): commencing operation
krb5kdc: starting...
W20260619 16:46:00.516335 392 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:46:00 dist-test-slave-03cd krb5kdc[22617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:46:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:46:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:46:00.656499 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38495
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38495,127.0.98.61:37271,127.0.98.60:44051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:00.782353 22630 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:46:00.782768 22630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:00.782823 22630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:00.782897 22630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:00.787365 22630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:00.787497 22630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:00.787535 22630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:00.787576 22630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:00.787616 22630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:00.794117 22630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38495,127.0.98.61:37271,127.0.98.60:44051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:00.795466 22630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:00.796703 22630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:00.806720 22635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:00.806726 22638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:00.806792 22630 server_base.cc:1034] running on GCE node
W20260619 16:46:00.806720 22636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:00.807824 22630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:00.809366 22630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:00.810691 22630 hybrid_clock.cc:648] HybridClock initialized: now 1781887560810602 us; error 177 us; skew 500 ppm
Jun 19 16:46:00 dist-test-slave-03cd krb5kdc[22617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:00.820104 22630 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:46:00.821440 22630 webserver.cc:460] Webserver started at http://127.0.98.62:36237/ using document root <none> and password file <none>
I20260619 16:46:00.822058 22630 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:00.822124 22630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:00.822309 22630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:00.824648 22630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "b319d3e41e6f4aacbf5f91aef259911e"
format_stamp: "Formatted at 2026-06-19 16:46:00 on dist-test-slave-03cd"
server_key: "c0d9454068cbfead128e9306d9035b25"
server_key_iv: "d4f2fa15187cd9b63045b8901d368de2"
server_key_version: "encryptionkey@0"
I20260619 16:46:00.825205 22630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "b319d3e41e6f4aacbf5f91aef259911e"
format_stamp: "Formatted at 2026-06-19 16:46:00 on dist-test-slave-03cd"
server_key: "c0d9454068cbfead128e9306d9035b25"
server_key_iv: "d4f2fa15187cd9b63045b8901d368de2"
server_key_version: "encryptionkey@0"
I20260619 16:46:00.829437 22630 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:46:00.832281 22645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:00.833772 22630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:00.833940 22630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "b319d3e41e6f4aacbf5f91aef259911e"
format_stamp: "Formatted at 2026-06-19 16:46:00 on dist-test-slave-03cd"
server_key: "c0d9454068cbfead128e9306d9035b25"
server_key_iv: "d4f2fa15187cd9b63045b8901d368de2"
server_key_version: "encryptionkey@0"
I20260619 16:46:00.834071 22630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:00.853645 22630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:00.857290 22630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:00.857470 22630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:00.864840 22630 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38495
I20260619 16:46:00.865139 22697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38495 every 8 connection(s)
I20260619 16:46:00.865921 22630 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:46:00.868968 22698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:00.872977 22698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38495 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:46:00.875168 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 22630
I20260619 16:46:00.875334 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:46:00.875648 392 external_mini_cluster.cc:1428] Setting key eaf36f6a42e1d48738a4b92cf329710f
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 19 16:46:00 dist-test-slave-03cd krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:46:00.897079 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:00.875820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38495 (local address 127.0.0.1:47508)
0619 16:46:00.876465 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:00.876527 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:00.877734 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:00.878699 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:00.878704 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:00.879089 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:00.879371 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:00.879386 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:00.881260 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:00.881262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:00.882051 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:00.882061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:00.883017 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:00.883020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:00.883119 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:00.883841 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:00.883859 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:00.889245 (+ 5386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:00.892587 (+ 3342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:00.892605 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:00.892621 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:00.893058 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:00.893626 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:00.893633 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:00.893638 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:00.893835 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:00.894695 (+ 860us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:00.894705 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:00.894941 (+ 236us) client_negotiation.cc:769] Sending connection context
0619 16:46:00.895200 (+ 259us) client_negotiation.cc:240] Negotiation successful
0619 16:46:00.895719 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"spinlock_wait_cycles":9472,"thread_start_us":181,"threads_started":1}
I20260619 16:46:00.897078 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:00.875820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47508 (local address 127.0.98.62:38495)
0619 16:46:00.876436 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:00.876454 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:00.876662 (+ 208us) server_negotiation.cc:407] Connection header received
0619 16:46:00.878148 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:00.878157 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:00.878447 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:00.878670 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:00.879536 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:00.881109 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:00.882193 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:00.882842 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:00.883043 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:00.889510 (+ 6467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:00.889522 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:00.889544 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:00.889585 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:00.892400 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:00.893345 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:00.893352 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:00.893353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:00.893499 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:00.893996 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:00.893999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:00.894001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:00.894541 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:00.894678 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:00.895475 (+ 797us) server_negotiation.cc:299] Negotiation successful
0619 16:46:00.896086 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":341,"server-negotiator.queue_time_us":490,"spinlock_wait_cycles":82304,"thread_start_us":181,"threads_started":1}
I20260619 16:46:00.900278 22698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37271 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:46:00.901266 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:00.900920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37271 (local address 127.0.0.1:37328)
0619 16:46:00.901106 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:00.901185 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260619 16:46:00.901523 22646 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37271: connect: Connection refused (error 111)
I20260619 16:46:00.902601 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:00.902310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37271 (local address 127.0.0.1:37342)
0619 16:46:00.902474 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:00.902524 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260619 16:46:00.902786 22698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37271: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37271: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 16:46:00.908540 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37271
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38495,127.0.98.61:37271,127.0.98.60:44051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:46:00.952095 22698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37271 } attempt: 1
I20260619 16:46:00.953135 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:00.952786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37271 (local address 127.0.0.1:37348)
0619 16:46:00.952991 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:00.953057 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 16:46:00.953898 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:00.953661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37271 (local address 127.0.0.1:37360)
0619 16:46:00.953789 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:00.953844 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 16:46:00.954033 22698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37271: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37271: connect: Connection refused (error 111)
I20260619 16:46:01.022260 22698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37271 } attempt: 2
I20260619 16:46:01.023255 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.022913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37271 (local address 127.0.0.1:37372)
0619 16:46:01.023103 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:01.023170 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260619 16:46:01.024078 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.023849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37271 (local address 127.0.0.1:37388)
0619 16:46:01.023960 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:01.024018 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 16:46:01.024215 22698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37271: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37271: connect: Connection refused (error 111)
W20260619 16:46:01.033941 22705 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:46:01.034363 22705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:01.034430 22705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:01.034526 22705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:01.038975 22705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:01.039113 22705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:01.039150 22705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:01.039186 22705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:01.039222 22705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:01.045240 22705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38495,127.0.98.61:37271,127.0.98.60:44051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:01.046494 22705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:01.047601 22705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:01.057087 22713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:01.057103 22711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:01.057111 22710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:01.057199 22705 server_base.cc:1034] running on GCE node
I20260619 16:46:01.057976 22705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:01.059211 22705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:01.060438 22705 hybrid_clock.cc:648] HybridClock initialized: now 1781887561060424 us; error 30 us; skew 500 ppm
Jun 19 16:46:01 dist-test-slave-03cd krb5kdc[22617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:01.064539 22705 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 16:46:01.065587 22705 webserver.cc:460] Webserver started at http://127.0.98.61:39535/ using document root <none> and password file <none>
I20260619 16:46:01.066150 22705 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:01.066219 22705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:01.066401 22705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:01.068606 22705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-1/data/instance:
uuid: "e70ddeb67b014e028c20279df3796f72"
format_stamp: "Formatted at 2026-06-19 16:46:01 on dist-test-slave-03cd"
server_key: "f0d87c7952318aa1852bfe7efed44609"
server_key_iv: "b1e40e353625e6eef77dbcf31cba4847"
server_key_version: "encryptionkey@0"
I20260619 16:46:01.069178 22705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-1/wal/instance:
uuid: "e70ddeb67b014e028c20279df3796f72"
format_stamp: "Formatted at 2026-06-19 16:46:01 on dist-test-slave-03cd"
server_key: "f0d87c7952318aa1852bfe7efed44609"
server_key_iv: "b1e40e353625e6eef77dbcf31cba4847"
server_key_version: "encryptionkey@0"
I20260619 16:46:01.073385 22705 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:46:01.076234 22720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:01.077776 22705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:01.077996 22705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-1/wal
uuid: "e70ddeb67b014e028c20279df3796f72"
format_stamp: "Formatted at 2026-06-19 16:46:01 on dist-test-slave-03cd"
server_key: "f0d87c7952318aa1852bfe7efed44609"
server_key_iv: "b1e40e353625e6eef77dbcf31cba4847"
server_key_version: "encryptionkey@0"
I20260619 16:46:01.078141 22705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:01.100531 22705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:01.104146 22705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:01.104393 22705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:01.112102 22705 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37271
I20260619 16:46:01.112481 22772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37271 every 8 connection(s)
I20260619 16:46:01.113166 22705 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-1/data/info.pb
I20260619 16:46:01.115401 22698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37271 } attempt: 3
I20260619 16:46:01.116446 22773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:01.116500 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 22705
I20260619 16:46:01.116609 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-1/wal/instance
I20260619 16:46:01.116936 392 external_mini_cluster.cc:1428] Setting key daf25653781ba08baf01d454d4fe6c23
I20260619 16:46:01.121641 22773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38495 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:46:01 dist-test-slave-03cd krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260619 16:46:01.129940 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.115981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37271 (local address 127.0.0.1:37404)
0619 16:46:01.116145 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:01.116164 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:01.116265 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:01.119178 (+ 2913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:01.119183 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:01.119409 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:01.119569 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:01.119578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.121036 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.121054 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:01.121866 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:01.121879 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.123023 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.123027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:01.123138 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:01.123886 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:01.123907 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:01.125589 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:01.127962 (+ 2373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:01.127966 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:01.127968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:01.128252 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:01.128625 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:01.128628 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:01.128630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:01.128718 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:01.129384 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:01.129387 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:01.129486 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 16:46:01.129610 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 16:46:01.129762 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 16:46:01 dist-test-slave-03cd krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:46:01.131130 22775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.116929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37404 (local address 127.0.98.61:37271)
0619 16:46:01.117487 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:01.117514 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:01.117553 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:46:01.118243 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:01.118260 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:01.118742 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:01.119206 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:01.119747 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.120865 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.122024 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.122866 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.123075 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:01.125786 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:01.125791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:01.125800 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:01.125825 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:01.127814 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:01.128373 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:01.128380 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:01.128386 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:01.128512 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:01.128812 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:01.128817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:01.128818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:01.129264 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:01.129415 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:01.129771 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 16:46:01.130123 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":177,"threads_started":1}
I20260619 16:46:01.133347 22698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44051 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:46:01.134523 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.134136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44051 (local address 127.0.0.1:38898)
0619 16:46:01.134329 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:01.134411 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 16:46:01.135607 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.135214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44051 (local address 127.0.0.1:38900)
0619 16:46:01.135442 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:01.135509 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
W20260619 16:46:01.135799 22698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44051: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44051: connect: Connection refused (error 111)
I20260619 16:46:01.138612 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.123841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47512 (local address 127.0.98.62:38495)
0619 16:46:01.123982 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:01.123987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:01.124369 (+ 382us) server_negotiation.cc:407] Connection header received
0619 16:46:01.124562 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:01.124567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:01.124620 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:01.124674 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:01.125717 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.126567 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.127609 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.128011 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.128132 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:01.133114 (+ 4982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:01.133117 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:01.133120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:01.133182 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:01.136226 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:01.137036 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:01.137039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:01.137041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:01.137123 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:01.137471 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:01.137474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:01.137475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:01.137677 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:01.137756 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:01.138300 (+ 544us) server_negotiation.cc:299] Negotiation successful
0619 16:46:01.138414 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:46:01.138825 22777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.123805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38495 (local address 127.0.0.1:47512)
0619 16:46:01.124231 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:01.124294 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:01.124455 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:01.124877 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:01.124886 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:01.125381 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:01.125568 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:01.125576 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.126712 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.126716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:01.127471 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:01.127481 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.128136 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.128139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:01.128280 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:01.128809 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:01.128821 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:01.132174 (+ 3353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:01.136384 (+ 4210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:01.136405 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:01.136415 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:01.136890 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:01.137233 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:01.137239 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:01.137241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:01.137364 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:01.137799 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:01.137818 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:01.138032 (+ 214us) client_negotiation.cc:769] Sending connection context
0619 16:46:01.138240 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 16:46:01.138530 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":141,"threads_started":1}
I20260619 16:46:01.140244 22773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37271 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:46:01 dist-test-slave-03cd krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 16:46:01.151422 22777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.140761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37271 (local address 127.0.0.1:37416)
0619 16:46:01.141017 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:01.141033 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:01.141117 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:01.141505 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:01.141507 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:01.141671 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:01.141805 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:01.141811 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.142749 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.142752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:01.143538 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:01.143549 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.144283 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.144286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:01.144403 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:01.145284 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:01.145298 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:01.146672 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:01.149375 (+ 2703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:01.149378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:01.149380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:01.149711 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:01.150150 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:01.150152 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:01.150154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:01.150243 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:01.150727 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:01.150729 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:01.150859 (+ 130us) client_negotiation.cc:769] Sending connection context
0619 16:46:01.151098 (+ 239us) client_negotiation.cc:240] Negotiation successful
0619 16:46:01.151273 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260619 16:46:01.152069 22775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.140878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37416 (local address 127.0.98.61:37271)
0619 16:46:01.141070 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:01.141076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:01.141284 (+ 208us) server_negotiation.cc:407] Connection header received
0619 16:46:01.141331 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:01.141334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:01.141374 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:01.141439 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:01.142052 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.142605 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.143763 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.144133 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.144233 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:01.147045 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:01.147049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:01.147052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:01.147085 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:01.149197 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:01.149853 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:01.149857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:01.149858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:01.149984 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:01.150399 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:01.150401 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:01.150403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:01.150610 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:01.150697 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:01.151791 (+ 1094us) server_negotiation.cc:299] Negotiation successful
0619 16:46:01.151946 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260619 16:46:01.152771 22773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44051 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:46:01.153731 22777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.153273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44051 (local address 127.0.0.1:38914)
0619 16:46:01.153472 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:01.153619 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260619 16:46:01.154457 22724 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44051: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 16:46:01.155778 22777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.155457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44051 (local address 127.0.0.1:38930)
0619 16:46:01.155600 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:01.155685 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 16:46:01.155998 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44051
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38495,127.0.98.61:37271,127.0.98.60:44051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:01.156037 22773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44051: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44051: connect: Connection refused (error 111)
I20260619 16:46:01.167028 22698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44051 } attempt: 1
I20260619 16:46:01.168069 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.167726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44051 (local address 127.0.0.1:38934)
0619 16:46:01.167923 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:01.167988 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 16:46:01.168985 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.168764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44051 (local address 127.0.0.1:38942)
0619 16:46:01.168875 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:01.168929 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 16:46:01.169121 22698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44051: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44051: connect: Connection refused (error 111)
I20260619 16:46:01.205448 22773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44051 } attempt: 1
I20260619 16:46:01.206564 22777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.206193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44051 (local address 127.0.0.1:38958)
0619 16:46:01.206406 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:01.206472 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260619 16:46:01.208053 22777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.207758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44051 (local address 127.0.0.1:38968)
0619 16:46:01.207924 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:01.207979 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260619 16:46:01.208395 22773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44051: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44051: connect: Connection refused (error 111)
I20260619 16:46:01.244272 22698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44051 } attempt: 2
I20260619 16:46:01.245174 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.244866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44051 (local address 127.0.0.1:38970)
0619 16:46:01.245014 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:01.245079 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 16:46:01.245921 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.245709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44051 (local address 127.0.0.1:38984)
0619 16:46:01.245817 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:01.245869 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 16:46:01.246105 22698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44051: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44051: connect: Connection refused (error 111)
I20260619 16:46:01.276721 22773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44051 } attempt: 2
I20260619 16:46:01.277659 22777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.277346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44051 (local address 127.0.0.1:38998)
0619 16:46:01.277497 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:01.277562 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 16:46:01.278471 22777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.278225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44051 (local address 127.0.0.1:39006)
0619 16:46:01.278335 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:01.278395 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 16:46:01.278718 22773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44051: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44051: connect: Connection refused (error 111)
W20260619 16:46:01.305604 22780 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:46:01.306147 22780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:01.306217 22780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:01.306304 22780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:01.311187 22780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:01.311337 22780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:01.311401 22780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:01.311446 22780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:01.311476 22780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:01.317813 22780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38495,127.0.98.61:37271,127.0.98.60:44051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:01.319285 22780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:01.320597 22780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:01.330422 22788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:01.330669 22786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:01.330816 22780 server_base.cc:1034] running on GCE node
W20260619 16:46:01.330415 22785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:01.331581 22780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:01.332854 22780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:01.334049 22780 hybrid_clock.cc:648] HybridClock initialized: now 1781887561334013 us; error 52 us; skew 500 ppm
Jun 19 16:46:01 dist-test-slave-03cd krb5kdc[22617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:01.337250 22780 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 16:46:01.338418 22780 webserver.cc:460] Webserver started at http://127.0.98.60:39949/ using document root <none> and password file <none>
I20260619 16:46:01.339390 22780 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:01.339515 22780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:01.339895 22780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:01.342303 22780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-2/data/instance:
uuid: "ff6cf28e325344c487ee5f3eebc892ef"
format_stamp: "Formatted at 2026-06-19 16:46:01 on dist-test-slave-03cd"
server_key: "4f811721dde4da6af05c1ee809b999a0"
server_key_iv: "0b13bb87a902f57370c49c82ca2895b1"
server_key_version: "encryptionkey@0"
I20260619 16:46:01.343014 22780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-2/wal/instance:
uuid: "ff6cf28e325344c487ee5f3eebc892ef"
format_stamp: "Formatted at 2026-06-19 16:46:01 on dist-test-slave-03cd"
server_key: "4f811721dde4da6af05c1ee809b999a0"
server_key_iv: "0b13bb87a902f57370c49c82ca2895b1"
server_key_version: "encryptionkey@0"
I20260619 16:46:01.345304 22698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44051 } attempt: 3
I20260619 16:46:01.346185 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.345843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44051 (local address 127.0.0.1:39012)
0619 16:46:01.346021 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:01.346095 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 16:46:01.347018 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.346780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44051 (local address 127.0.0.1:39024)
0619 16:46:01.346902 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:01.346952 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 16:46:01.347223 22698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44051: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44051: connect: Connection refused (error 111)
I20260619 16:46:01.347837 22780 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260619 16:46:01.350831 22795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:01.352510 22780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:01.352686 22780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-2/wal
uuid: "ff6cf28e325344c487ee5f3eebc892ef"
format_stamp: "Formatted at 2026-06-19 16:46:01 on dist-test-slave-03cd"
server_key: "4f811721dde4da6af05c1ee809b999a0"
server_key_iv: "0b13bb87a902f57370c49c82ca2895b1"
server_key_version: "encryptionkey@0"
I20260619 16:46:01.352837 22780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:01.367412 22780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:01.369896 22773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44051 } attempt: 3
I20260619 16:46:01.370869 22777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.370530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44051 (local address 127.0.0.1:39038)
0619 16:46:01.370720 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:01.370781 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 16:46:01.371843 22780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:01.371944 22777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.371606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44051 (local address 127.0.0.1:39042)
0619 16:46:01.371797 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:01.371849 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260619 16:46:01.372058 22780 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260619 16:46:01.372174 22773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44051: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44051: connect: Connection refused (error 111)
I20260619 16:46:01.381024 22780 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44051
I20260619 16:46:01.381366 22847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44051 every 8 connection(s)
I20260619 16:46:01.382366 22780 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-2/data/info.pb
I20260619 16:46:01.384200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 22780
I20260619 16:46:01.384301 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887259574980-392-0/minicluster-data/master-2/wal/instance
I20260619 16:46:01.384617 392 external_mini_cluster.cc:1428] Setting key 65ab3d0bf7cef040da7634c22393b38a
I20260619 16:46:01.386163 22848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:01.391359 22848 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38495 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:46:01 dist-test-slave-03cd krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 16:46:01 dist-test-slave-03cd krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:46:01.401849 22851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.386052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39054 (local address 127.0.98.60:44051)
0619 16:46:01.386513 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:01.386539 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:01.386583 (+ 44us) server_negotiation.cc:407] Connection header received
0619 16:46:01.387463 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:01.387486 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:01.387992 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:01.388382 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:01.389330 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.390863 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.392038 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.393041 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.393222 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:01.396522 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:01.396529 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:01.396543 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:01.396583 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:01.398650 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:01.399187 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:01.399194 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:01.399200 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:01.399346 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:01.399680 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:01.399685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:01.399688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:01.400184 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:01.400438 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:01.400665 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 16:46:01.401012 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":159,"threads_started":1}
I20260619 16:46:01.405403 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.394116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38495 (local address 127.0.0.1:47526)
0619 16:46:01.394671 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:01.394722 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:01.394925 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:01.395336 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:01.395343 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:01.395818 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:01.396042 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:01.396052 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.397131 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.397135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:01.397850 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:01.397861 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.398417 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.398422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:01.398704 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:01.399542 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:01.399555 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:01.401300 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:01.403335 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:01.403344 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:01.403350 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:01.403605 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:01.404027 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:01.404033 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:01.404038 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:01.404137 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:01.404653 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:01.404662 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:01.404794 (+ 132us) client_negotiation.cc:769] Sending connection context
0619 16:46:01.404946 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 16:46:01.405163 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":206,"threads_started":1}
I20260619 16:46:01.405440 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.394158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47526 (local address 127.0.98.62:38495)
0619 16:46:01.394312 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:01.394316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:01.394848 (+ 532us) server_negotiation.cc:407] Connection header received
0619 16:46:01.395052 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:01.395056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:01.395116 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:01.395218 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:01.396192 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.396994 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.398019 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.398279 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.398414 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:01.401483 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:01.401487 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:01.401489 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:01.401516 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:01.403207 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:01.403776 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:01.403783 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:01.403788 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:01.403907 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:01.404257 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:01.404264 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:01.404269 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:01.404530 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:01.404650 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:01.405043 (+ 393us) server_negotiation.cc:299] Negotiation successful
0619 16:46:01.405190 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 16:46:01.406877 22848 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37271 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:46:01 dist-test-slave-03cd krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 16:46:01 dist-test-slave-03cd krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 16:46:01.415513 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.404753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47536 (local address 127.0.98.62:38495)
0619 16:46:01.405131 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:01.405144 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:01.405157 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:46:01.405192 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:01.405194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:01.405237 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:01.406477 (+ 1240us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:01.406579 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.407233 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.408168 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.408536 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.408671 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:01.411821 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:01.411825 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:01.411827 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:01.411855 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:01.413683 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:01.414162 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:01.414166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:01.414168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:01.414256 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:01.414591 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:01.414596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:01.414598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:01.414801 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:01.415031 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:01.415153 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 16:46:01.415293 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":121,"threads_started":1}
I20260619 16:46:01.415697 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.407436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37271 (local address 127.0.0.1:37432)
0619 16:46:01.407727 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:01.407743 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:01.407826 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:01.408213 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:01.408216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:01.408379 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:01.408490 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:01.408496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.409271 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.409274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:01.409780 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:01.409789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.410499 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.410503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:01.410624 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:01.411087 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:01.411105 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:01.412482 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:01.414119 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:01.414123 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:01.414125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:01.414327 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:01.414644 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:01.414648 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:01.414650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:01.414738 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:01.415123 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:01.415127 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:01.415227 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 16:46:01.415334 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 16:46:01.415490 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198}
I20260619 16:46:01.416214 22775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.407579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37432 (local address 127.0.98.61:37271)
0619 16:46:01.407730 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:01.407736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:01.407785 (+ 49us) server_negotiation.cc:407] Connection header received
0619 16:46:01.407950 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:01.407953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:01.408057 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:01.408197 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:01.408607 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.409144 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.409924 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.410319 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.410406 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:01.412637 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:01.412641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:01.412643 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:01.412668 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:01.414000 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:01.414463 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:01.414468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:01.414470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:01.414548 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:01.414837 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:01.414840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:01.414842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:01.415019 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:01.415122 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:01.415395 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 16:46:01.415508 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 16:46:01.416966 22848 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44051 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:46:01 dist-test-slave-03cd krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 16:46:01 dist-test-slave-03cd krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 16:46:01.426263 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.417505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44051 (local address 127.0.0.1:39056)
0619 16:46:01.417697 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:01.417713 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:01.417790 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:01.418021 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:01.418026 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:01.418192 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:01.418267 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:01.418273 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.419008 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.419014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:01.419514 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:01.419521 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.420052 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.420055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:01.420128 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:01.420576 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:01.420589 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:01.422115 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:01.424519 (+ 2404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:01.424522 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:01.424524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:01.424725 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:01.425070 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:01.425072 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:01.425074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:01.425158 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:01.425623 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:01.425629 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:01.425746 (+ 117us) client_negotiation.cc:769] Sending connection context
0619 16:46:01.425836 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 16:46:01.426020 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260619 16:46:01.426359 22851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.417630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39056 (local address 127.0.98.60:44051)
0619 16:46:01.417787 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:01.417790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:01.417802 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:46:01.417874 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:01.417876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:01.417908 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:01.417977 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:01.418381 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.418901 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.419630 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.419940 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.420052 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:01.422320 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:01.422325 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:01.422328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:01.422357 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:01.424374 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:01.424835 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:01.424841 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:01.424845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:01.424939 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:01.425260 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:01.425263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:01.425264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:01.425482 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:01.425573 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:01.425955 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 16:46:01.426076 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:46:01.427318 22775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.416822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37440 (local address 127.0.98.61:37271)
0619 16:46:01.416927 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:01.416931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:01.416997 (+ 66us) server_negotiation.cc:407] Connection header received
0619 16:46:01.417137 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:01.417139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:01.417184 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:01.417267 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:01.418082 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.418577 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.419387 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.419616 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.419783 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:01.423174 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:01.423180 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:01.423185 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:01.423242 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:01.424941 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:01.425717 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:01.425720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:01.425721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:01.425806 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:01.426147 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:01.426152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:01.426153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:01.426402 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:01.426666 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:01.427095 (+ 429us) server_negotiation.cc:299] Negotiation successful
0619 16:46:01.427202 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 16:46:01.430266 22848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff6cf28e325344c487ee5f3eebc892ef: Bootstrap starting.
I20260619 16:46:01.431598 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:46:01.432899 22848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff6cf28e325344c487ee5f3eebc892ef: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:01.433707 22848 log.cc:824] T 00000000000000000000000000000000 P ff6cf28e325344c487ee5f3eebc892ef: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:01.433120 22672 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b319d3e41e6f4aacbf5f91aef259911e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47536
I20260619 16:46:01.435863 22848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff6cf28e325344c487ee5f3eebc892ef: No bootstrap required, opened a new log
I20260619 16:46:01.440007 22848 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff6cf28e325344c487ee5f3eebc892ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b319d3e41e6f4aacbf5f91aef259911e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38495 } } peers { permanent_uuid: "e70ddeb67b014e028c20279df3796f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37271 } } peers { permanent_uuid: "ff6cf28e325344c487ee5f3eebc892ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44051 } }
I20260619 16:46:01.440315 22848 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff6cf28e325344c487ee5f3eebc892ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6cf28e325344c487ee5f3eebc892ef, State: Initialized, Role: FOLLOWER
I20260619 16:46:01.440709 22848 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff6cf28e325344c487ee5f3eebc892ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b319d3e41e6f4aacbf5f91aef259911e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38495 } } peers { permanent_uuid: "e70ddeb67b014e028c20279df3796f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37271 } } peers { permanent_uuid: "ff6cf28e325344c487ee5f3eebc892ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44051 } }
I20260619 16:46:01.441308 22855 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff6cf28e325344c487ee5f3eebc892ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b319d3e41e6f4aacbf5f91aef259911e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38495 } } peers { permanent_uuid: "e70ddeb67b014e028c20279df3796f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37271 } } peers { permanent_uuid: "ff6cf28e325344c487ee5f3eebc892ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44051 } } }
I20260619 16:46:01.441461 22855 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff6cf28e325344c487ee5f3eebc892ef [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:46:01.442130 22848 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff6cf28e325344c487ee5f3eebc892ef [sys.catalog]: configured and running, proceeding with master startup.
W20260619 16:46:01.447831 22866 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ff6cf28e325344c487ee5f3eebc892ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:46:01.447957 22866 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:46:01.446449 22780 master_runner.cc:186] Error getting master registration for 127.0.98.62:38495: OK, instance_id { permanent_uuid: "b319d3e41e6f4aacbf5f91aef259911e" instance_seqno: 1781887560862648 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:46:01.450191 22780 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:46:01.511423 22698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44051 } attempt: 4
I20260619 16:46:01.515348 22773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44051 } attempt: 4
Jun 19 16:46:01 dist-test-slave-03cd krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 16:46:01.522640 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.512161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44051 (local address 127.0.0.1:39058)
0619 16:46:01.512350 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:01.512371 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:01.512484 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:01.512806 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:01.512809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:01.513046 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:01.513149 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:01.513155 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.514059 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.514065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:01.514651 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:01.514661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.515203 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.515207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:01.515303 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:01.515953 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:01.515972 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:01.517952 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:01.520504 (+ 2552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:01.520508 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:01.520510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:01.521009 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:01.521387 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:01.521390 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:01.521392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:01.521488 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:01.522048 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:01.522051 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:01.522161 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 16:46:01.522263 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 16:46:01.522431 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 16:46:01.522640 22851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.512243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39058 (local address 127.0.98.60:44051)
0619 16:46:01.512389 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:01.512393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:01.512417 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:46:01.512622 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:01.512626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:01.512685 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:01.512785 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:01.513321 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.513921 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.514802 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.515063 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.515191 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:01.518115 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:01.518118 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:01.518121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:01.518144 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:01.520179 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:01.521164 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:01.521169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:01.521173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:01.521274 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:01.521598 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:01.521602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:01.521604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:01.521912 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:01.522001 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:01.522343 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 16:46:01.522473 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 16:46:01 dist-test-slave-03cd krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 16:46:01.527735 22698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b319d3e41e6f4aacbf5f91aef259911e: Bootstrap starting.
I20260619 16:46:01.528765 22777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.515929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44051 (local address 127.0.0.1:39072)
0619 16:46:01.516169 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:01.516186 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:01.516294 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:01.517385 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:01.517389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:01.517615 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:01.517887 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:01.517898 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.519461 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.519467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:01.520369 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:01.520385 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.521795 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.521799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:01.521896 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:01.522476 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:01.522492 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:01.524296 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:01.526664 (+ 2368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:01.526668 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:01.526670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:01.526980 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:01.527453 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:01.527457 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:01.527459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:01.527568 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:01.528170 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:01.528173 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:01.528299 (+ 126us) client_negotiation.cc:769] Sending connection context
0619 16:46:01.528397 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 16:46:01.528565 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260619 16:46:01.529799 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.516584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39072 (local address 127.0.98.60:44051)
0619 16:46:01.517046 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:01.517067 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:01.517087 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:46:01.517132 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:01.517138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:01.517200 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:01.518049 (+ 849us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:01.518180 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.519051 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.521186 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.521622 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.521785 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:01.524462 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:01.524466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:01.524471 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:01.524501 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:01.526459 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:01.527139 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:01.527143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:01.527145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:01.527318 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:01.527746 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:01.527751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:01.527756 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:01.528035 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:01.529131 (+ 1096us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:01.529242 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 16:46:01.529399 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":367,"thread_start_us":144,"threads_started":1}
I20260619 16:46:01.531049 22698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b319d3e41e6f4aacbf5f91aef259911e: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:01.532055 22698 log.cc:824] T 00000000000000000000000000000000 P b319d3e41e6f4aacbf5f91aef259911e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:01.533531 22773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e70ddeb67b014e028c20279df3796f72: Bootstrap starting.
I20260619 16:46:01.534687 22698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b319d3e41e6f4aacbf5f91aef259911e: No bootstrap required, opened a new log
I20260619 16:46:01.536410 22773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e70ddeb67b014e028c20279df3796f72: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:01.537356 22773 log.cc:824] T 00000000000000000000000000000000 P e70ddeb67b014e028c20279df3796f72: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:01.538278 22698 raft_consensus.cc:348] T 00000000000000000000000000000000 P b319d3e41e6f4aacbf5f91aef259911e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b319d3e41e6f4aacbf5f91aef259911e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38495 } } peers { permanent_uuid: "e70ddeb67b014e028c20279df3796f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37271 } } peers { permanent_uuid: "ff6cf28e325344c487ee5f3eebc892ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44051 } }
I20260619 16:46:01.538569 22698 raft_consensus.cc:729] T 00000000000000000000000000000000 P b319d3e41e6f4aacbf5f91aef259911e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b319d3e41e6f4aacbf5f91aef259911e, State: Initialized, Role: FOLLOWER
I20260619 16:46:01.538956 22698 consensus_queue.cc:260] T 00000000000000000000000000000000 P b319d3e41e6f4aacbf5f91aef259911e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b319d3e41e6f4aacbf5f91aef259911e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38495 } } peers { permanent_uuid: "e70ddeb67b014e028c20279df3796f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37271 } } peers { permanent_uuid: "ff6cf28e325344c487ee5f3eebc892ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44051 } }
I20260619 16:46:01.539649 22871 sys_catalog.cc:455] T 00000000000000000000000000000000 P b319d3e41e6f4aacbf5f91aef259911e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b319d3e41e6f4aacbf5f91aef259911e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38495 } } peers { permanent_uuid: "e70ddeb67b014e028c20279df3796f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37271 } } peers { permanent_uuid: "ff6cf28e325344c487ee5f3eebc892ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44051 } } }
I20260619 16:46:01.540050 22773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e70ddeb67b014e028c20279df3796f72: No bootstrap required, opened a new log
I20260619 16:46:01.540030 22871 sys_catalog.cc:458] T 00000000000000000000000000000000 P b319d3e41e6f4aacbf5f91aef259911e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:46:01.540557 22698 sys_catalog.cc:564] T 00000000000000000000000000000000 P b319d3e41e6f4aacbf5f91aef259911e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:01.540623 22672 raft_consensus.cc:482] T 00000000000000000000000000000000 P b319d3e41e6f4aacbf5f91aef259911e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 16:46:01.540725 22672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b319d3e41e6f4aacbf5f91aef259911e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:01.542369 22672 raft_consensus.cc:504] T 00000000000000000000000000000000 P b319d3e41e6f4aacbf5f91aef259911e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b319d3e41e6f4aacbf5f91aef259911e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38495 } } peers { permanent_uuid: "e70ddeb67b014e028c20279df3796f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37271 } } peers { permanent_uuid: "ff6cf28e325344c487ee5f3eebc892ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44051 } }
I20260619 16:46:01.544641 22672 leader_election.cc:288] T 00000000000000000000000000000000 P b319d3e41e6f4aacbf5f91aef259911e [CANDIDATE]: Term 1 election: Requested vote from peers e70ddeb67b014e028c20279df3796f72 (127.0.98.61:37271), ff6cf28e325344c487ee5f3eebc892ef (127.0.98.60:44051)
I20260619 16:46:01.545336 22821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b319d3e41e6f4aacbf5f91aef259911e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ff6cf28e325344c487ee5f3eebc892ef"
I20260619 16:46:01.545626 22821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff6cf28e325344c487ee5f3eebc892ef [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:01.547178 22821 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ff6cf28e325344c487ee5f3eebc892ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b319d3e41e6f4aacbf5f91aef259911e in term 1.
I20260619 16:46:01.546689 22630 master_runner.cc:186] Error getting master registration for 127.0.98.61:37271: OK, instance_id { permanent_uuid: "e70ddeb67b014e028c20279df3796f72" instance_seqno: 1781887561109998 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:46:01.550100 22630 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:46:01.551080 22773 raft_consensus.cc:348] T 00000000000000000000000000000000 P e70ddeb67b014e028c20279df3796f72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b319d3e41e6f4aacbf5f91aef259911e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38495 } } peers { permanent_uuid: "e70ddeb67b014e028c20279df3796f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37271 } } peers { permanent_uuid: "ff6cf28e325344c487ee5f3eebc892ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44051 } }
I20260619 16:46:01.551355 22773 raft_consensus.cc:729] T 00000000000000000000000000000000 P e70ddeb67b014e028c20279df3796f72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e70ddeb67b014e028c20279df3796f72, State: Initialized, Role: FOLLOWER
I20260619 16:46:01.551748 22773 consensus_queue.cc:260] T 00000000000000000000000000000000 P e70ddeb67b014e028c20279df3796f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b319d3e41e6f4aacbf5f91aef259911e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38495 } } peers { permanent_uuid: "e70ddeb67b014e028c20279df3796f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37271 } } peers { permanent_uuid: "ff6cf28e325344c487ee5f3eebc892ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44051 } }
I20260619 16:46:01.553273 22773 sys_catalog.cc:564] T 00000000000000000000000000000000 P e70ddeb67b014e028c20279df3796f72 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:01.553321 22747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b319d3e41e6f4aacbf5f91aef259911e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e70ddeb67b014e028c20279df3796f72"
I20260619 16:46:01.553598 22747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e70ddeb67b014e028c20279df3796f72 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:01.555154 22747 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e70ddeb67b014e028c20279df3796f72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b319d3e41e6f4aacbf5f91aef259911e in term 1.
W20260619 16:46:01.547217 22882 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b319d3e41e6f4aacbf5f91aef259911e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260619 16:46:01.547855 22649 leader_election.cc:302] T 00000000000000000000000000000000 P b319d3e41e6f4aacbf5f91aef259911e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b319d3e41e6f4aacbf5f91aef259911e, ff6cf28e325344c487ee5f3eebc892ef; no voters:
I20260619 16:46:01.555871 22871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b319d3e41e6f4aacbf5f91aef259911e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:01.556113 22871 raft_consensus.cc:686] T 00000000000000000000000000000000 P b319d3e41e6f4aacbf5f91aef259911e [term 1 LEADER]: Becoming Leader. State: Replica: b319d3e41e6f4aacbf5f91aef259911e, State: Running, Role: LEADER
I20260619 16:46:01.556509 22871 consensus_queue.cc:237] T 00000000000000000000000000000000 P b319d3e41e6f4aacbf5f91aef259911e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b319d3e41e6f4aacbf5f91aef259911e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38495 } } peers { permanent_uuid: "e70ddeb67b014e028c20279df3796f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37271 } } peers { permanent_uuid: "ff6cf28e325344c487ee5f3eebc892ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44051 } }
W20260619 16:46:01.558864 22882 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:46:01.560635 22902 sys_catalog.cc:455] T 00000000000000000000000000000000 P b319d3e41e6f4aacbf5f91aef259911e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b319d3e41e6f4aacbf5f91aef259911e. Latest consensus state: current_term: 1 leader_uuid: "b319d3e41e6f4aacbf5f91aef259911e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b319d3e41e6f4aacbf5f91aef259911e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38495 } } peers { permanent_uuid: "e70ddeb67b014e028c20279df3796f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37271 } } peers { permanent_uuid: "ff6cf28e325344c487ee5f3eebc892ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44051 } } }
I20260619 16:46:01.560834 22902 sys_catalog.cc:458] T 00000000000000000000000000000000 P b319d3e41e6f4aacbf5f91aef259911e [sys.catalog]: This master's current role is: LEADER
W20260619 16:46:01.567559 22900 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e70ddeb67b014e028c20279df3796f72: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:46:01.567708 22900 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:46:01.570124 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.552805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47538 (local address 127.0.98.62:38495)
0619 16:46:01.552952 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:01.552957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:01.557830 (+ 4873us) server_negotiation.cc:407] Connection header received
0619 16:46:01.557880 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:01.557883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:01.557932 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:01.558028 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:01.561301 (+ 3273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.562125 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.563353 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.563752 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.563843 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:01.565906 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:01.565911 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:01.565914 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:01.565945 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:01.567988 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:01.568579 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:01.568584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:01.568585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:01.568682 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:01.568980 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:01.568983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:01.568985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:01.569213 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:01.569330 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:01.569807 (+ 477us) server_negotiation.cc:299] Negotiation successful
0619 16:46:01.569956 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:46:01.571957 22905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:46:01.572821 22905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:46:01.578392 22821 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ff6cf28e325344c487ee5f3eebc892ef [term 1 FOLLOWER]: Refusing update from remote peer b319d3e41e6f4aacbf5f91aef259911e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:01.579308 22902 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b319d3e41e6f4aacbf5f91aef259911e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6cf28e325344c487ee5f3eebc892ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:01.581352 22889 sys_catalog.cc:455] T 00000000000000000000000000000000 P e70ddeb67b014e028c20279df3796f72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b319d3e41e6f4aacbf5f91aef259911e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38495 } } peers { permanent_uuid: "e70ddeb67b014e028c20279df3796f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37271 } } peers { permanent_uuid: "ff6cf28e325344c487ee5f3eebc892ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44051 } } }
I20260619 16:46:01.581534 22889 sys_catalog.cc:458] T 00000000000000000000000000000000 P e70ddeb67b014e028c20279df3796f72 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:46:01.584286 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.551005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39088 (local address 127.0.98.60:44051)
0619 16:46:01.551148 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:01.551153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:01.557353 (+ 6200us) server_negotiation.cc:407] Connection header received
0619 16:46:01.557586 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:01.557590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:01.557656 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:01.557721 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:01.559738 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.560538 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.569187 (+ 8649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.569568 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.569664 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:01.577039 (+ 7375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:01.577045 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:01.577049 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:01.577088 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:01.579339 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:01.581770 (+ 2431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:01.581778 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:01.581782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:01.581932 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:01.582500 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:01.582504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:01.582506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:01.582795 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:01.582955 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:01.583892 (+ 937us) server_negotiation.cc:299] Negotiation successful
0619 16:46:01.584057 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:46:01.584997 22855 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff6cf28e325344c487ee5f3eebc892ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader b319d3e41e6f4aacbf5f91aef259911e. Latest consensus state: current_term: 1 leader_uuid: "b319d3e41e6f4aacbf5f91aef259911e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b319d3e41e6f4aacbf5f91aef259911e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38495 } } peers { permanent_uuid: "e70ddeb67b014e028c20279df3796f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37271 } } peers { permanent_uuid: "ff6cf28e325344c487ee5f3eebc892ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44051 } } }
I20260619 16:46:01.585157 22855 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff6cf28e325344c487ee5f3eebc892ef [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:46:01.585903 22747 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e70ddeb67b014e028c20279df3796f72 [term 1 FOLLOWER]: Refusing update from remote peer b319d3e41e6f4aacbf5f91aef259911e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:01.586633 22902 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b319d3e41e6f4aacbf5f91aef259911e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e70ddeb67b014e028c20279df3796f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:01.592046 22889 sys_catalog.cc:455] T 00000000000000000000000000000000 P e70ddeb67b014e028c20279df3796f72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b319d3e41e6f4aacbf5f91aef259911e. Latest consensus state: current_term: 1 leader_uuid: "b319d3e41e6f4aacbf5f91aef259911e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b319d3e41e6f4aacbf5f91aef259911e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38495 } } peers { permanent_uuid: "e70ddeb67b014e028c20279df3796f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37271 } } peers { permanent_uuid: "ff6cf28e325344c487ee5f3eebc892ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44051 } } }
I20260619 16:46:01.592206 22889 sys_catalog.cc:458] T 00000000000000000000000000000000 P e70ddeb67b014e028c20279df3796f72 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:46:01.593626 22775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:01.556702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37448 (local address 127.0.98.61:37271)
0619 16:46:01.557048 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:01.557052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:01.558915 (+ 1863us) server_negotiation.cc:407] Connection header received
0619 16:46:01.568041 (+ 9126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:01.568045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:01.568099 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:01.568158 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:01.570685 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.571434 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.574062 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:01.574449 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:01.574540 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:01.587949 (+ 13409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:01.587956 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:01.587963 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:01.588000 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:01.590384 (+ 2384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:01.591270 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:01.591274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:01.591276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:01.591376 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:01.592567 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:01.592571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:01.592573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:01.592841 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:01.592959 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:01.593322 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 16:46:01.593458 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259}
I20260619 16:46:01.595532 22871 sys_catalog.cc:455] T 00000000000000000000000000000000 P b319d3e41e6f4aacbf5f91aef259911e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b319d3e41e6f4aacbf5f91aef259911e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b319d3e41e6f4aacbf5f91aef259911e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38495 } } peers { permanent_uuid: "e70ddeb67b014e028c20279df3796f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37271 } } peers { permanent_uuid: "ff6cf28e325344c487ee5f3eebc892ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44051 } } }
I20260619 16:46:01.595719 22871 sys_catalog.cc:458] T 00000000000000000000000000000000 P b319d3e41e6f4aacbf5f91aef259911e [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:01.595870 22907 mvcc.cc:205] Tried to move back new op lower bound from 7298611452216455168 to 7298611452140187648. Current Snapshot: MvccSnapshot[applied={T|T < 7298611452216455168}]
I20260619 16:46:01.596084 22855 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff6cf28e325344c487ee5f3eebc892ef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b319d3e41e6f4aacbf5f91aef259911e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b319d3e41e6f4aacbf5f91aef259911e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38495 } } peers { permanent_uuid: "e70ddeb67b014e028c20279df3796f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37271 } } peers { permanent_uuid: "ff6cf28e325344c487ee5f3eebc892ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44051 } } }
I20260619 16:46:01.596207 22855 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff6cf28e325344c487ee5f3eebc892ef [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:46:01.599274 22902 sys_catalog.cc:455] T 00000000000000000000000000000000 P b319d3e41e6f4aacbf5f91aef259911e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b319d3e41e6f4aacbf5f91aef259911e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b319d3e41e6f4aacbf5f91aef259911e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38495 } } peers { permanent_uuid: "e70ddeb67b014e028c20279df3796f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37271 } } peers { permanent_uuid: "ff6cf28e325344c487ee5f3eebc892ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44051 } } }
I20260619 16:46:01.599436 22902 sys_catalog.cc:458] T 00000000000000000000000000000000 P b319d3e41e6f4aacbf5f91aef259911e [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:01.601532 22889 sys_catalog.cc:455] T 00000000000000000000000000000000 P e70ddeb67b014e028c20279df3796f72 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b319d3e41e6f4aacbf5f91aef259911e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b319d3e41e6f4aacbf5f91aef259911e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38495 } } peers { permanent_uuid: "e70ddeb67b014e028c20279df3796f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37271 } } peers { permanent_uuid: "ff6cf28e325344c487ee5f3eebc892ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44051 } } }
I20260619 16:46:01.601866 22889 sys_catalog.cc:458] T 00000000000000000000000000000000 P e70ddeb67b014e028c20279df3796f72 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:46:01.603693 22910 mvcc.cc:205] Tried to move back new op lower bound from 7298611452216455168 to 7298611452140187648. Current Snapshot: MvccSnapshot[applied={T|T < 7298611452216455168}]
I20260619 16:46:01.608249 22905 catalog_manager.cc:1347] Generated new cluster ID: 683bf208785643b98eb268d821df0d10
I20260619 16:46:01.608364 22905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:46:01.635694 22905 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:46:01.637218 22905 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:46:01.654383 22905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b319d3e41e6f4aacbf5f91aef259911e: Generated new TSK 0
I20260619 16:46:01.655385 22905 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:46:01.690778 22775 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:01.689119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37462 (local address 127.0.98.61:37271)
0619 16:46:01.689258 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:01.689264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:01.689621 (+ 357us) server_negotiation.cc:407] Connection header received
0619 16:46:01.689826 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:01.689830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:01.689881 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:01.689946 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:01.690633 (+ 687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37462: BlockingRecv error: recv got EOF from 127.0.0.1:37462 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 16:46:01.691856 22853 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:01.688261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47550 (local address 127.0.98.62:38495)
0619 16:46:01.688399 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:01.688403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:01.689125 (+ 722us) server_negotiation.cc:407] Connection header received
0619 16:46:01.690166 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:01.690171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:01.690228 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:01.690287 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:01.691711 (+ 1424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47550: BlockingRecv error: recv got EOF from 127.0.0.1:47550 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 16:46:01.692308 22868 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:01.688263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39094 (local address 127.0.98.60:44051)
0619 16:46:01.688401 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:01.688406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:01.689597 (+ 1191us) server_negotiation.cc:407] Connection header received
0619 16:46:01.690439 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:01.690442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:01.690500 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:01.690559 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:01.692186 (+ 1627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39094: BlockingRecv error: recv got EOF from 127.0.0.1:39094 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 16:46:01.696703 22775 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:01.695256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37478 (local address 127.0.98.61:37271)
0619 16:46:01.695381 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:01.695386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:01.695622 (+ 236us) server_negotiation.cc:407] Connection header received
0619 16:46:01.695870 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:01.695873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:01.695968 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:01.696061 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:01.696578 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37478: BlockingRecv error: recv got EOF from 127.0.0.1:37478 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 16:46:01.701297 22775 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:01.700314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37482 (local address 127.0.98.61:37271)
0619 16:46:01.700433 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:01.700438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:01.700453 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:46:01.700513 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:01.700516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:01.700570 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:01.700672 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:01.701170 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37482: BlockingRecv error: recv got EOF from 127.0.0.1:37482 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 16:46:01.706789 22775 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:01.705525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37490 (local address 127.0.98.61:37271)
0619 16:46:01.705652 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:01.705659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:01.705870 (+ 211us) server_negotiation.cc:407] Connection header received
0619 16:46:01.706009 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:01.706012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:01.706069 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:01.706166 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:01.706647 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37490: BlockingRecv error: recv got EOF from 127.0.0.1:37490 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 16:46:01.714771 22775 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:01.713299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37506 (local address 127.0.98.61:37271)
0619 16:46:01.713437 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:01.713442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:01.713691 (+ 249us) server_negotiation.cc:407] Connection header received
0619 16:46:01.713863 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:01.713867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:01.713937 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:01.714050 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:01.714643 (+ 593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37506: BlockingRecv error: recv got EOF from 127.0.0.1:37506 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 16:46:01.726809 22775 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:01.725474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37516 (local address 127.0.98.61:37271)
0619 16:46:01.725618 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:01.725623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:01.725790 (+ 167us) server_negotiation.cc:407] Connection header received
0619 16:46:01.725938 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:01.725942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:01.725999 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:01.726105 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:01.726673 (+ 568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37516: BlockingRecv error: recv got EOF from 127.0.0.1:37516 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 16:46:01.746976 22775 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:01.745538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37530 (local address 127.0.98.61:37271)
0619 16:46:01.745671 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:01.745676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:01.745939 (+ 263us) server_negotiation.cc:407] Connection header received
0619 16:46:01.746091 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:01.746094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:01.746153 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:01.746257 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:01.746846 (+ 589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37530: BlockingRecv error: recv got EOF from 127.0.0.1:37530 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 16:46:01.782966 22775 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:01.781501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37538 (local address 127.0.98.61:37271)
0619 16:46:01.781641 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:01.781647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:01.781877 (+ 230us) server_negotiation.cc:407] Connection header received
0619 16:46:01.782037 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:01.782041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:01.782103 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:01.782230 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:01.782822 (+ 592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37538: BlockingRecv error: recv got EOF from 127.0.0.1:37538 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 16:46:01.850504 22775 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:01.849249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37542 (local address 127.0.98.61:37271)
0619 16:46:01.849418 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:01.849424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:01.849579 (+ 155us) server_negotiation.cc:407] Connection header received
0619 16:46:01.849750 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:01.849753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:01.849818 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:01.849915 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:01.850376 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37542: BlockingRecv error: recv got EOF from 127.0.0.1:37542 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 16:46:01.982352 22775 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:01.980926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37556 (local address 127.0.98.61:37271)
0619 16:46:01.981141 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:01.981146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:01.981295 (+ 149us) server_negotiation.cc:407] Connection header received
0619 16:46:01.981494 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:01.981501 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:01.981551 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:01.981648 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:01.982203 (+ 555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37556: BlockingRecv error: recv got EOF from 127.0.0.1:37556 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
W20260619 16:46:02.243616 22775 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:02.241468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37562 (local address 127.0.98.61:37271)
0619 16:46:02.241725 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:02.241732 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:02.241939 (+ 207us) server_negotiation.cc:407] Connection header received
0619 16:46:02.242426 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:02.242430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:02.242560 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:02.242695 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:02.243469 (+ 774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37562: BlockingRecv error: recv got EOF from 127.0.0.1:37562 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
I20260619 16:46:02.449434 22866 catalog_manager.cc:1259] Loaded cluster ID: 683bf208785643b98eb268d821df0d10
I20260619 16:46:02.449545 22866 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ff6cf28e325344c487ee5f3eebc892ef: loading cluster ID for follower catalog manager: success
I20260619 16:46:02.451294 22866 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ff6cf28e325344c487ee5f3eebc892ef: acquiring CA information for follower catalog manager: success
I20260619 16:46:02.452551 22866 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ff6cf28e325344c487ee5f3eebc892ef: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:46:02.569160 22900 catalog_manager.cc:1259] Loaded cluster ID: 683bf208785643b98eb268d821df0d10
I20260619 16:46:02.569262 22900 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e70ddeb67b014e028c20279df3796f72: loading cluster ID for follower catalog manager: success
I20260619 16:46:02.571084 22900 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e70ddeb67b014e028c20279df3796f72: acquiring CA information for follower catalog manager: success
I20260619 16:46:02.572484 22900 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e70ddeb67b014e028c20279df3796f72: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:46:02.763775 22979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:02.758268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37566 (local address 127.0.98.61:37271)
0619 16:46:02.758780 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:02.758784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:02.758811 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:46:02.758952 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:02.758955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:02.759105 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:02.759238 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:02.759822 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:02.760388 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:02.761779 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:02.762048 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:02.762164 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:02.762660 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:02.763143 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:02.763260 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:02.763461 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 16:46:02.763530 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":143,"threads_started":1}
I20260619 16:46:02.773548 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:02.768630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39108 (local address 127.0.98.60:44051)
0619 16:46:02.769031 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:02.769038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:02.769064 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:46:02.769179 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:02.769182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:02.769342 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:02.769501 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:02.769782 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:02.770422 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:02.771475 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:02.771843 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:02.771981 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:02.772390 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:02.773018 (+ 628us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:02.773189 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:02.773237 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 16:46:02.773319 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":119,"threads_started":1}
I20260619 16:46:02.783790 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:02.778828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39110 (local address 127.0.98.60:44051)
0619 16:46:02.779091 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:02.779097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:02.779247 (+ 150us) server_negotiation.cc:407] Connection header received
0619 16:46:02.780010 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:02.780014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:02.780087 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:02.780194 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:02.780512 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:02.781109 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:02.782410 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:02.782772 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:02.782894 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:02.783208 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:02.783318 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:02.783403 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:02.783508 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 16:46:02.783580 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260619 16:46:02.784741 22979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:02.779324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37572 (local address 127.0.98.61:37271)
0619 16:46:02.779454 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:02.779457 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:02.779470 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:46:02.779523 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:02.779526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:02.779575 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:02.779724 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:02.780047 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:02.781010 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:02.782963 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:02.783363 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:02.783455 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:02.784119 (+ 664us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:02.784221 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:02.784284 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:02.784499 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 16:46:02.784594 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 16:46:02.785280 22991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:02.778510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47562 (local address 127.0.98.62:38495)
0619 16:46:02.778853 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:02.778860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:02.779183 (+ 323us) server_negotiation.cc:407] Connection header received
0619 16:46:02.779548 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:02.779551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:02.779801 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:02.779967 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:02.780316 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:02.781252 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:02.782963 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:02.783363 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:02.783469 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:02.784119 (+ 650us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:02.784547 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:02.784641 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:02.785018 (+ 377us) server_negotiation.cc:299] Negotiation successful
0619 16:46:02.785101 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":96,"threads_started":1}
I20260619 16:46:02.789837 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 22630
I20260619 16:46:02.797178 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 22705
I20260619 16:46:02.803794 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 22780
2026-06-19T16:46:02Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3372 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:46:02 dist-test-slave-03cd krb5kdc[22996](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:46:02 dist-test-slave-03cd krb5kdc[22996](info): set up 2 sockets
Jun 19 16:46:02 dist-test-slave-03cd krb5kdc[22996](info): commencing operation
krb5kdc: starting...
W20260619 16:46:03.880296 392 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/krb5kdc/test-user.keytab.
Jun 19 16:46:03 dist-test-slave-03cd krb5kdc[22996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:46:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:46:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:46:04.020767 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40257
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40257,127.0.98.61:46705,127.0.98.60:40553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:04.143254 23009 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:46:04.143657 23009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:04.143750 23009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:04.143869 23009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:04.148229 23009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:04.148341 23009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:04.148377 23009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:04.148419 23009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:04.148455 23009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:04.154199 23009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40257,127.0.98.61:46705,127.0.98.60:40553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:04.155467 23009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:04.156646 23009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:04.166276 23015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:04.166317 23017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:04.166314 23014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:04.166381 23009 server_base.cc:1034] running on GCE node
I20260619 16:46:04.167338 23009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:04.168810 23009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:04.170035 23009 hybrid_clock.cc:648] HybridClock initialized: now 1781887564170013 us; error 39 us; skew 500 ppm
Jun 19 16:46:04 dist-test-slave-03cd krb5kdc[22996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:04.173420 23009 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:46:04.174816 23009 webserver.cc:460] Webserver started at http://127.0.98.62:44671/ using document root <none> and password file <none>
I20260619 16:46:04.175638 23009 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:04.175838 23009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:04.176110 23009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:04.178566 23009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-0/data/instance:
uuid: "98b75f7c127143dca5a81f6708400c37"
format_stamp: "Formatted at 2026-06-19 16:46:04 on dist-test-slave-03cd"
server_key: "5dbe1d6bc9a3a35968b1ea34635a2b5b"
server_key_iv: "781cd538f6a36930bd20357b775a2bdc"
server_key_version: "encryptionkey@0"
I20260619 16:46:04.179245 23009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-0/wal/instance:
uuid: "98b75f7c127143dca5a81f6708400c37"
format_stamp: "Formatted at 2026-06-19 16:46:04 on dist-test-slave-03cd"
server_key: "5dbe1d6bc9a3a35968b1ea34635a2b5b"
server_key_iv: "781cd538f6a36930bd20357b775a2bdc"
server_key_version: "encryptionkey@0"
I20260619 16:46:04.183444 23009 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 16:46:04.186605 23024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:04.188136 23009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:04.188377 23009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-0/wal
uuid: "98b75f7c127143dca5a81f6708400c37"
format_stamp: "Formatted at 2026-06-19 16:46:04 on dist-test-slave-03cd"
server_key: "5dbe1d6bc9a3a35968b1ea34635a2b5b"
server_key_iv: "781cd538f6a36930bd20357b775a2bdc"
server_key_version: "encryptionkey@0"
I20260619 16:46:04.188537 23009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:04.215828 23009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:04.220057 23009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:04.220333 23009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:04.229035 23009 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40257
I20260619 16:46:04.229468 23076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40257 every 8 connection(s)
I20260619 16:46:04.230218 23009 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:46:04.233450 23077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:04.237718 23077 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40257 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:46:04.238982 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 23009
I20260619 16:46:04.239099 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-0/wal/instance
I20260619 16:46:04.239390 392 external_mini_cluster.cc:1428] Setting key 77943741e3898973429bc01e49700171
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 19 16:46:04 dist-test-slave-03cd krb5kdc[22996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:46:04.259366 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.240931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40257 (local address 127.0.0.1:48226)
0619 16:46:04.241671 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:04.241737 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:04.243224 (+ 1487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:04.244223 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:04.244232 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:04.244817 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:04.245281 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:04.245305 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.246534 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.246538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:04.248014 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:04.248032 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.248840 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.248845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:04.249011 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:04.249716 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:04.249736 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:04.253296 (+ 3560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:04.255723 (+ 2427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:04.255736 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:04.255750 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:04.256096 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:04.256587 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:04.256590 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:04.256592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:04.256750 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:04.257551 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:04.257561 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:04.257768 (+ 207us) client_negotiation.cc:769] Sending connection context
0619 16:46:04.257992 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 16:46:04.258493 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":104,"spinlock_wait_cycles":60288,"thread_start_us":187,"threads_started":1}
I20260619 16:46:04.259384 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.240931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48226 (local address 127.0.98.62:40257)
0619 16:46:04.241546 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:04.241569 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:04.241873 (+ 304us) server_negotiation.cc:407] Connection header received
0619 16:46:04.243587 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:04.243592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:04.243993 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:04.244154 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:04.245475 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.246388 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.248189 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.248600 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.248818 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:04.253512 (+ 4694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:04.253523 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:04.253544 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:04.253581 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:04.255545 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.256289 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.256300 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.256302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.256467 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.256915 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.256918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.256922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.257412 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:04.257531 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:04.258109 (+ 578us) server_negotiation.cc:299] Negotiation successful
0619 16:46:04.258382 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":417,"thread_start_us":187,"threads_started":1}
I20260619 16:46:04.262217 23077 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46705 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:46:04.262992 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.262727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46705 (local address 127.0.0.1:52558)
0619 16:46:04.262838 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:04.262920 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 16:46:04.263197 23027 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46705: connect: Connection refused (error 111)
I20260619 16:46:04.264235 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.263962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46705 (local address 127.0.0.1:52572)
0619 16:46:04.264081 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:04.264163 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 16:46:04.264395 23077 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46705: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46705: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 16:46:04.272300 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46705
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40257,127.0.98.61:46705,127.0.98.60:40553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:46:04.313700 23077 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46705 } attempt: 1
I20260619 16:46:04.314631 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.314330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46705 (local address 127.0.0.1:52588)
0619 16:46:04.314486 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:04.314555 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 16:46:04.315549 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.315298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46705 (local address 127.0.0.1:52598)
0619 16:46:04.315435 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:04.315486 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 16:46:04.315898 23077 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46705: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46705: connect: Connection refused (error 111)
I20260619 16:46:04.384088 23077 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46705 } attempt: 2
I20260619 16:46:04.384997 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.384688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46705 (local address 127.0.0.1:52610)
0619 16:46:04.384836 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:04.384910 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 16:46:04.385910 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.385643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46705 (local address 127.0.0.1:52626)
0619 16:46:04.385781 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:04.385843 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 16:46:04.386286 23077 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46705: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46705: connect: Connection refused (error 111)
W20260619 16:46:04.392486 23084 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:46:04.392827 23084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:04.392854 23084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:04.392957 23084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:04.397923 23084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:04.398032 23084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:04.398057 23084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:04.398100 23084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:04.398138 23084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:04.405159 23084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40257,127.0.98.61:46705,127.0.98.60:40553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:04.406425 23084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:04.407564 23084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:04.416831 23089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:04.416785 23092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:04.416885 23090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:04.417042 23084 server_base.cc:1034] running on GCE node
I20260619 16:46:04.417804 23084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:04.419167 23084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:04.420348 23084 hybrid_clock.cc:648] HybridClock initialized: now 1781887564420304 us; error 65 us; skew 500 ppm
Jun 19 16:46:04 dist-test-slave-03cd krb5kdc[22996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:04.423491 23084 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 16:46:04.424607 23084 webserver.cc:460] Webserver started at http://127.0.98.61:41273/ using document root <none> and password file <none>
I20260619 16:46:04.425228 23084 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:04.425297 23084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:04.425540 23084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:04.427837 23084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-1/data/instance:
uuid: "54f560782b1848f698f3bfe3d63c490c"
format_stamp: "Formatted at 2026-06-19 16:46:04 on dist-test-slave-03cd"
server_key: "6a39ec3e0c18d628f22b0afc44e9b8ef"
server_key_iv: "2fae0b7b62b70d067302349a48404bfe"
server_key_version: "encryptionkey@0"
I20260619 16:46:04.428443 23084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-1/wal/instance:
uuid: "54f560782b1848f698f3bfe3d63c490c"
format_stamp: "Formatted at 2026-06-19 16:46:04 on dist-test-slave-03cd"
server_key: "6a39ec3e0c18d628f22b0afc44e9b8ef"
server_key_iv: "2fae0b7b62b70d067302349a48404bfe"
server_key_version: "encryptionkey@0"
I20260619 16:46:04.432389 23084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 16:46:04.435071 23099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:04.436517 23084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:46:04.436676 23084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-1/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-1/wal
uuid: "54f560782b1848f698f3bfe3d63c490c"
format_stamp: "Formatted at 2026-06-19 16:46:04 on dist-test-slave-03cd"
server_key: "6a39ec3e0c18d628f22b0afc44e9b8ef"
server_key_iv: "2fae0b7b62b70d067302349a48404bfe"
server_key_version: "encryptionkey@0"
I20260619 16:46:04.436808 23084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:04.455930 23084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:04.459055 23084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:04.459285 23084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:04.466430 23084 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46705
I20260619 16:46:04.466717 23151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46705 every 8 connection(s)
I20260619 16:46:04.467525 23084 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-1/data/info.pb
I20260619 16:46:04.469871 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 23084
I20260619 16:46:04.469996 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-1/wal/instance
I20260619 16:46:04.470314 392 external_mini_cluster.cc:1428] Setting key 4013c6142632fc02d80120d66ec392c5
I20260619 16:46:04.470883 23152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:04.475533 23152 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40257 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:46:04.477494 23077 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46705 } attempt: 3
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 16:46:04 dist-test-slave-03cd krb5kdc[22996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 16:46:04 dist-test-slave-03cd krb5kdc[22996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 16:46:04.491648 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.477895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46705 (local address 127.0.0.1:52634)
0619 16:46:04.478046 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:04.478060 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:04.478162 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:04.480497 (+ 2335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:04.480501 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:04.480729 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:04.480869 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:04.480878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.482468 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.482474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:04.483564 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:04.483579 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.484348 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.484354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:04.484470 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:04.484938 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:04.484960 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:04.486750 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:04.489588 (+ 2838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:04.489592 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:04.489594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:04.489880 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:04.490274 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:04.490276 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:04.490278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:04.490361 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:04.490977 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:04.490983 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:04.491105 (+ 122us) client_negotiation.cc:769] Sending connection context
0619 16:46:04.491243 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 16:46:04.491420 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":92}
I20260619 16:46:04.491866 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.477806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48232 (local address 127.0.98.62:40257)
0619 16:46:04.477957 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:04.477962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:04.479117 (+ 1155us) server_negotiation.cc:407] Connection header received
0619 16:46:04.480199 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:04.480202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:04.480271 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:04.480382 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:04.481339 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.482067 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.483195 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.483779 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.483916 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:04.487044 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:04.487050 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:04.487054 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:04.487078 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:04.488845 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.489454 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.489457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.489459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.489540 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.489889 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.489893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.489894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.490088 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:04.490192 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:04.491484 (+ 1292us) server_negotiation.cc:299] Negotiation successful
0619 16:46:04.491613 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":110,"server-negotiator.queue_time_us":62}
I20260619 16:46:04.492516 23155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.478325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40257 (local address 127.0.0.1:48232)
0619 16:46:04.478972 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:04.479015 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:04.479998 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:04.480430 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:04.480439 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:04.480921 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:04.481198 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:04.481215 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.482204 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.482208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:04.483047 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:04.483057 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.483947 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.483952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:04.484114 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:04.484804 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:04.484827 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:04.486903 (+ 2076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:04.488990 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:04.488999 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:04.489010 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:04.489342 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:04.489644 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:04.489647 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:04.489649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:04.489790 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:04.490205 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:04.490214 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:04.490574 (+ 360us) client_negotiation.cc:769] Sending connection context
0619 16:46:04.490769 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 16:46:04.491176 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":3,"thread_start_us":161,"threads_started":1}
I20260619 16:46:04.492573 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.479314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52634 (local address 127.0.98.61:46705)
0619 16:46:04.479615 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:04.479625 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:04.479738 (+ 113us) server_negotiation.cc:407] Connection header received
0619 16:46:04.480012 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:04.480035 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:04.480302 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:04.480525 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:04.481000 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.482321 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.483700 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.484199 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.484321 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:04.486919 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:04.486925 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:04.486936 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:04.486985 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:04.489440 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.490054 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.490062 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.490067 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.490183 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.490465 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.490469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.490471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.490817 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:04.492119 (+ 1302us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:04.492262 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 16:46:04.492420 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":196,"thread_start_us":121,"threads_started":1}
I20260619 16:46:04.494252 23152 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46705 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:46:04.494803 23077 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40553 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:46:04.495779 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.495409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40553 (local address 127.0.0.1:55248)
0619 16:46:04.495553 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:04.495619 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 16:46:04.496657 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.496379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40553 (local address 127.0.0.1:55262)
0619 16:46:04.496533 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:04.496589 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260619 16:46:04.496833 23077 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40553: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40553: connect: Connection refused (error 111)
Jun 19 16:46:04 dist-test-slave-03cd krb5kdc[22996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 16:46:04.504485 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8rFIPy/build/debug/bin/kudu
/tmp/dist-test-task8rFIPy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40553
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40257,127.0.98.61:46705,127.0.98.60:40553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:46:04.507083 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.495177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52642 (local address 127.0.98.61:46705)
0619 16:46:04.495334 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:04.495337 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:04.495350 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:46:04.495404 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:04.495407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:04.495470 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:04.495555 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:04.496154 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.496994 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.497824 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.498270 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.498359 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:04.501778 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:04.501782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:04.501784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:04.501810 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:04.504221 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.505038 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.505046 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.505051 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.505186 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.505627 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.505634 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.505639 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.506074 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:04.506237 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:04.506669 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 16:46:04.506901 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":73}
I20260619 16:46:04.507097 23155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.495002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46705 (local address 127.0.0.1:52642)
0619 16:46:04.495139 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:04.495156 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:04.495238 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:04.495585 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:04.495588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:04.495804 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:04.495928 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:04.495935 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.497090 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.497093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:04.497678 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:04.497686 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.498526 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.498532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:04.498750 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:04.499508 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:04.499526 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:04.501588 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:04.504462 (+ 2874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:04.504467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:04.504468 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:04.504868 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:04.505384 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:04.505387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:04.505388 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:04.505495 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:04.506227 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:04.506230 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:04.506398 (+ 168us) client_negotiation.cc:769] Sending connection context
0619 16:46:04.506558 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 16:46:04.506791 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 16:46:04.508216 23152 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40553 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:46:04.509404 23155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.508801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40553 (local address 127.0.0.1:55270)
0619 16:46:04.509140 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:04.509294 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260619 16:46:04.509784 23103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40553: connect: Connection refused (error 111)
I20260619 16:46:04.510994 23155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.510717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40553 (local address 127.0.0.1:55284)
0619 16:46:04.510862 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:04.510917 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 16:46:04.511324 23152 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40553: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40553: connect: Connection refused (error 111)
I20260619 16:46:04.527997 23077 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40553 } attempt: 1
I20260619 16:46:04.529081 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.528724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40553 (local address 127.0.0.1:55300)
0619 16:46:04.528903 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:04.528980 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 16:46:04.529909 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.529660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40553 (local address 127.0.0.1:55306)
0619 16:46:04.529784 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:04.529839 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 16:46:04.530094 23077 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40553: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40553: connect: Connection refused (error 111)
I20260619 16:46:04.560629 23152 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40553 } attempt: 1
I20260619 16:46:04.561702 23155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.561363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40553 (local address 127.0.0.1:55318)
0619 16:46:04.561558 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:04.561621 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 16:46:04.562435 23155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.562224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40553 (local address 127.0.0.1:55326)
0619 16:46:04.562328 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:04.562383 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 16:46:04.562613 23152 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40553: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40553: connect: Connection refused (error 111)
I20260619 16:46:04.605315 23077 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40553 } attempt: 2
I20260619 16:46:04.606267 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.605901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40553 (local address 127.0.0.1:55336)
0619 16:46:04.606084 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:04.606165 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 16:46:04.607100 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.606861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40553 (local address 127.0.0.1:55352)
0619 16:46:04.606981 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:04.607035 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 16:46:04.607280 23077 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40553: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40553: connect: Connection refused (error 111)
W20260619 16:46:04.625896 23158 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:46:04.626295 23158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:04.626381 23158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:04.626456 23158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:04.630828 23158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:04.630932 23158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
I20260619 16:46:04.630826 23152 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40553 } attempt: 2
W20260619 16:46:04.630964 23158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:04.630987 23158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:04.631008 23158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:04.631791 23155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.631390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40553 (local address 127.0.0.1:55366)
0619 16:46:04.631558 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:04.631643 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 16:46:04.632570 23155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.632344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40553 (local address 127.0.0.1:55382)
0619 16:46:04.632454 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:04.632512 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 16:46:04.632731 23152 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40553: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40553: connect: Connection refused (error 111)
I20260619 16:46:04.637468 23158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40257,127.0.98.61:46705,127.0.98.60:40553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:04.638780 23158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:04.640058 23158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:04.650130 23164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:04.650136 23165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:04.650168 23158 server_base.cc:1034] running on GCE node
W20260619 16:46:04.650187 23167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:04.651262 23158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:04.652658 23158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:04.653864 23158 hybrid_clock.cc:648] HybridClock initialized: now 1781887564653846 us; error 38 us; skew 500 ppm
Jun 19 16:46:04 dist-test-slave-03cd krb5kdc[22996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:04.657279 23158 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 16:46:04.658329 23158 webserver.cc:460] Webserver started at http://127.0.98.60:37407/ using document root <none> and password file <none>
I20260619 16:46:04.658931 23158 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:04.658994 23158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:04.659231 23158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:04.661619 23158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-2/data/instance:
uuid: "8c1c16e258ad4cf885edcdd117bc8503"
format_stamp: "Formatted at 2026-06-19 16:46:04 on dist-test-slave-03cd"
server_key: "da25d31b144895292105e5958cab9851"
server_key_iv: "4f08ff10adb8c92944028a00eb762923"
server_key_version: "encryptionkey@0"
I20260619 16:46:04.662258 23158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-2/wal/instance:
uuid: "8c1c16e258ad4cf885edcdd117bc8503"
format_stamp: "Formatted at 2026-06-19 16:46:04 on dist-test-slave-03cd"
server_key: "da25d31b144895292105e5958cab9851"
server_key_iv: "4f08ff10adb8c92944028a00eb762923"
server_key_version: "encryptionkey@0"
I20260619 16:46:04.666801 23158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260619 16:46:04.669806 23174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:04.671337 23158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 16:46:04.671563 23158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-2/data,/tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-2/wal
uuid: "8c1c16e258ad4cf885edcdd117bc8503"
format_stamp: "Formatted at 2026-06-19 16:46:04 on dist-test-slave-03cd"
server_key: "da25d31b144895292105e5958cab9851"
server_key_iv: "4f08ff10adb8c92944028a00eb762923"
server_key_version: "encryptionkey@0"
I20260619 16:46:04.671772 23158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:04.689424 23158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:04.693802 23158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:04.694046 23158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:04.702014 23158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40553
I20260619 16:46:04.702299 23226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40553 every 8 connection(s)
I20260619 16:46:04.703146 23158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-2/data/info.pb
I20260619 16:46:04.704391 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8rFIPy/build/debug/bin/kudu as pid 23158
I20260619 16:46:04.704522 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8rFIPy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887259574980-392-0/minicluster-data/master-2/wal/instance
I20260619 16:46:04.704825 392 external_mini_cluster.cc:1428] Setting key f00ff9313e62bf030b2fcfbfa681b27b
I20260619 16:46:04.706508 23077 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40553 } attempt: 3
I20260619 16:46:04.707647 23227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:04.712440 23227 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40257 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:46:04 dist-test-slave-03cd krb5kdc[22996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 16:46:04 dist-test-slave-03cd krb5kdc[22996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 16:46:04.721887 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.707019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40553 (local address 127.0.0.1:55392)
0619 16:46:04.707155 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:04.707171 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:04.707271 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:04.709105 (+ 1834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:04.709108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:04.709438 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:04.709558 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:04.709568 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.711659 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.711663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:04.712477 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:04.712492 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.714028 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.714032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:04.714130 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:04.714738 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:04.714757 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:04.716500 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:04.719528 (+ 3028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:04.719532 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:04.719534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:04.719859 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:04.720275 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:04.720279 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:04.720281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:04.720385 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:04.721236 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:04.721240 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:04.721365 (+ 125us) client_negotiation.cc:769] Sending connection context
0619 16:46:04.721495 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 16:46:04.721689 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 16:46:04.723157 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.706401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55390 (local address 127.0.98.60:40553)
0619 16:46:04.707003 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:04.707025 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:04.707073 (+ 48us) server_negotiation.cc:407] Connection header received
0619 16:46:04.708049 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:04.708075 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:04.708587 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:04.708977 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:04.709894 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.711494 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.712526 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.713440 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.713667 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:04.717305 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:04.717309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:04.717313 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:04.717346 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:04.719424 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.720157 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.720160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.720162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.720283 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.720670 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.720673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.720674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.721114 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:04.722373 (+ 1259us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:04.722473 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 16:46:04.722591 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":859,"server-negotiator.queue_time_us":399,"spinlock_wait_cycles":17152,"thread_start_us":158,"threads_started":1}
I20260619 16:46:04.723176 23231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.707089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55392 (local address 127.0.98.60:40553)
0619 16:46:04.707421 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:04.707429 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:04.707443 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:46:04.708049 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:04.708075 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:04.708587 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:04.708977 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:04.709738 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.711494 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.712594 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.713440 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.713664 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:04.716727 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:04.716734 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:04.716747 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:04.716786 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:04.719384 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.719985 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.719992 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.720001 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.720151 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.720530 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.720534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.720537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.721095 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:04.721280 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:04.721684 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 16:46:04.722045 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":463,"server-negotiator.queue_time_us":253,"spinlock_wait_cycles":14464,"thread_start_us":126,"threads_started":1}
Jun 19 16:46:04 dist-test-slave-03cd krb5kdc[22996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:46:04.723889 23152 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40553 } attempt: 3
I20260619 16:46:04.728257 23077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98b75f7c127143dca5a81f6708400c37: Bootstrap starting.
I20260619 16:46:04.728821 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.714753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48238 (local address 127.0.98.62:40257)
0619 16:46:04.714890 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:04.714894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:04.715252 (+ 358us) server_negotiation.cc:407] Connection header received
0619 16:46:04.715655 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:04.715684 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:04.715735 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:04.715839 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:04.716744 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.717535 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.719086 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.719471 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.719576 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:04.724144 (+ 4568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:04.724149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:04.724152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:04.724183 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:04.726123 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.726792 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.726796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.726798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.726899 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.727276 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.727281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.727282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.727584 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:04.727909 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:04.728477 (+ 568us) server_negotiation.cc:299] Negotiation successful
0619 16:46:04.728621 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:46:04.729880 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.714558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40257 (local address 127.0.0.1:48238)
0619 16:46:04.715131 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:04.715183 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:04.715516 (+ 333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:04.715897 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:04.715903 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:04.716416 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:04.716606 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:04.716615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.717851 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.717854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:04.718921 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:04.718939 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.719734 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.719737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:04.720007 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:04.720691 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:04.720741 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:04.723953 (+ 3212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:04.726267 (+ 2314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:04.726281 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:04.726289 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:04.726655 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:04.727036 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:04.727041 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:04.727043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:04.727150 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:04.728021 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:04.728031 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:04.728202 (+ 171us) client_negotiation.cc:769] Sending connection context
0619 16:46:04.729389 (+ 1187us) client_negotiation.cc:240] Negotiation successful
0619 16:46:04.729651 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"spinlock_wait_cycles":6016,"thread_start_us":155,"threads_started":1}
Jun 19 16:46:04 dist-test-slave-03cd krb5kdc[22996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 16:46:04.731231 23227 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46705 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:46:04.731995 23077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98b75f7c127143dca5a81f6708400c37: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:04.733536 23077 log.cc:824] T 00000000000000000000000000000000 P 98b75f7c127143dca5a81f6708400c37: Log is configured to *not* fsync() on all Append() calls
Jun 19 16:46:04 dist-test-slave-03cd krb5kdc[22996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:46:04.736013 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.725031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55398 (local address 127.0.98.60:40553)
0619 16:46:04.725136 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:04.725140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:04.725151 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:46:04.725468 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:04.725471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:04.725519 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:04.726116 (+ 597us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:04.726201 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.726946 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.728180 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.728549 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.729025 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:04.731258 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:04.731261 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:04.731264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:04.731292 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:04.733730 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.734476 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.734479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.734481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.734576 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.734943 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.734948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.734952 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.735177 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:04.735277 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:04.735692 (+ 415us) server_negotiation.cc:299] Negotiation successful
0619 16:46:04.735822 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 16:46:04.736289 23155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.724544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40553 (local address 127.0.0.1:55398)
0619 16:46:04.724793 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:04.724808 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:04.724888 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:04.725655 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:04.725659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:04.725875 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:04.726001 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:04.726009 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.727083 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.727086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:04.728017 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:04.728029 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.728694 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.728698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:04.728783 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:04.729320 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:04.729334 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:04.731084 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:04.733858 (+ 2774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:04.733863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:04.733865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:04.734266 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:04.734699 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:04.734702 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:04.734704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:04.734813 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:04.735324 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:04.735328 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:04.735445 (+ 117us) client_negotiation.cc:769] Sending connection context
0619 16:46:04.735586 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 16:46:04.735789 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260619 16:46:04.737697 23077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98b75f7c127143dca5a81f6708400c37: No bootstrap required, opened a new log
Jun 19 16:46:04 dist-test-slave-03cd krb5kdc[22996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 16:46:04.740152 23152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54f560782b1848f698f3bfe3d63c490c: Bootstrap starting.
I20260619 16:46:04.742512 23077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98b75f7c127143dca5a81f6708400c37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b75f7c127143dca5a81f6708400c37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40257 } } peers { permanent_uuid: "54f560782b1848f698f3bfe3d63c490c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46705 } } peers { permanent_uuid: "8c1c16e258ad4cf885edcdd117bc8503" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40553 } }
I20260619 16:46:04.742779 23077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98b75f7c127143dca5a81f6708400c37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98b75f7c127143dca5a81f6708400c37, State: Initialized, Role: FOLLOWER
I20260619 16:46:04.743068 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.732437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52652 (local address 127.0.98.61:46705)
0619 16:46:04.732553 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:04.732558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:04.732573 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:46:04.732618 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:04.732622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:04.732672 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:04.732769 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:04.733378 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.734213 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.735442 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.735843 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.735953 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:04.738952 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:04.738958 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:04.738962 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:04.738990 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:04.740842 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.741565 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.741569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.741571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.741680 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.742010 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.742013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.742015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.742289 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:04.742362 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:04.742767 (+ 405us) server_negotiation.cc:299] Negotiation successful
0619 16:46:04.742906 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":35}
I20260619 16:46:04.743165 23077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98b75f7c127143dca5a81f6708400c37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b75f7c127143dca5a81f6708400c37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40257 } } peers { permanent_uuid: "54f560782b1848f698f3bfe3d63c490c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46705 } } peers { permanent_uuid: "8c1c16e258ad4cf885edcdd117bc8503" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40553 } }
I20260619 16:46:04.743569 23152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54f560782b1848f698f3bfe3d63c490c: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:04.744001 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.732122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46705 (local address 127.0.0.1:52652)
0619 16:46:04.732373 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:04.732391 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:04.732475 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:04.732822 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:04.732828 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:04.733037 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:04.733209 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:04.733220 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.734440 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.734443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:04.735324 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:04.735333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.736072 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.736075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:04.736273 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:04.737160 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:04.737174 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:04.738782 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:04.741123 (+ 2341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:04.741127 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:04.741130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:04.741415 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:04.741804 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:04.741807 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:04.741809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:04.741902 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:04.742450 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:04.742453 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:04.742559 (+ 106us) client_negotiation.cc:769] Sending connection context
0619 16:46:04.743687 (+ 1128us) client_negotiation.cc:240] Negotiation successful
0619 16:46:04.743851 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260619 16:46:04.744354 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.726460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48254 (local address 127.0.98.62:40257)
0619 16:46:04.726810 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:04.726824 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:04.727188 (+ 364us) server_negotiation.cc:407] Connection header received
0619 16:46:04.727236 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:04.727239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:04.727295 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:04.728670 (+ 1375us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:04.728773 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.729920 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.731465 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.731913 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.732147 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:04.735558 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:04.735561 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:04.735564 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:04.735590 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:04.741597 (+ 6007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.742296 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.742300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.742302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.742406 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.742768 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.742771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.742773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.743035 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:04.743919 (+ 884us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:04.744015 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 16:46:04.744152 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":101,"threads_started":1}
I20260619 16:46:04.744885 23077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98b75f7c127143dca5a81f6708400c37 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:04.744983 23227 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40553 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:46:04.745275 23235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98b75f7c127143dca5a81f6708400c37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b75f7c127143dca5a81f6708400c37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40257 } } peers { permanent_uuid: "54f560782b1848f698f3bfe3d63c490c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46705 } } peers { permanent_uuid: "8c1c16e258ad4cf885edcdd117bc8503" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40553 } } }
I20260619 16:46:04.745594 23235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98b75f7c127143dca5a81f6708400c37 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:46:04.745288 23152 log.cc:824] T 00000000000000000000000000000000 P 54f560782b1848f698f3bfe3d63c490c: Log is configured to *not* fsync() on all Append() calls
W20260619 16:46:04.751233 23246 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 98b75f7c127143dca5a81f6708400c37: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:46:04.751420 23246 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 19 16:46:04 dist-test-slave-03cd krb5kdc[22996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 16:46:04 dist-test-slave-03cd krb5kdc[22996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 16:46:04.750551 23009 master_runner.cc:186] Error getting master registration for 127.0.98.61:46705: OK, instance_id { permanent_uuid: "54f560782b1848f698f3bfe3d63c490c" instance_seqno: 1781887564464386 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:46:04.754966 23009 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:46:04.756204 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.745512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40553 (local address 127.0.0.1:55402)
0619 16:46:04.745625 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:04.745641 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:04.745731 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:04.746440 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:04.746442 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:04.746642 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:04.746776 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:04.746786 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.747936 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.747939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:04.748809 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:04.748819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.749504 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.749509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:04.749586 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:04.750117 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:04.750133 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:04.752051 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:04.754236 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:04.754239 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:04.754242 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:04.754509 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:04.754837 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:04.754841 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:04.754847 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:04.754952 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:04.755510 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:04.755516 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:04.755629 (+ 113us) client_negotiation.cc:769] Sending connection context
0619 16:46:04.755790 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 16:46:04.755956 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 16:46:04.756862 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.746008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55402 (local address 127.0.98.60:40553)
0619 16:46:04.746121 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:04.746126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:04.746140 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:46:04.746189 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:04.746192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:04.746242 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:04.746326 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:04.746929 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.747822 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.748932 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.749289 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.749377 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:04.752218 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:04.752225 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:04.752229 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:04.752260 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:04.754085 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.754631 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.754635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.754638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.754728 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.755065 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.755068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.755070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.755364 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:04.755469 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:04.756553 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0619 16:46:04.756675 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 16:46:04.757510 23152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54f560782b1848f698f3bfe3d63c490c: No bootstrap required, opened a new log
I20260619 16:46:04.759588 23227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c1c16e258ad4cf885edcdd117bc8503: Bootstrap starting.
I20260619 16:46:04.761408 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.746355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52660 (local address 127.0.98.61:46705)
0619 16:46:04.746481 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:04.746485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:04.746501 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:46:04.746542 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:04.746545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:04.746589 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:04.746689 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:04.747506 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.748356 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.749530 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.750249 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.750365 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:04.754284 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:04.754292 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:04.754297 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:04.754330 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:04.756256 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.756833 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.756837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.756839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.756957 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.757478 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.757482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.757484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.760054 (+ 2570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:04.760289 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:04.761052 (+ 763us) server_negotiation.cc:299] Negotiation successful
0619 16:46:04.761217 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:46:04.762614 23152 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54f560782b1848f698f3bfe3d63c490c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b75f7c127143dca5a81f6708400c37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40257 } } peers { permanent_uuid: "54f560782b1848f698f3bfe3d63c490c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46705 } } peers { permanent_uuid: "8c1c16e258ad4cf885edcdd117bc8503" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40553 } }
I20260619 16:46:04.762913 23152 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54f560782b1848f698f3bfe3d63c490c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f560782b1848f698f3bfe3d63c490c, State: Initialized, Role: FOLLOWER
I20260619 16:46:04.762960 23227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c1c16e258ad4cf885edcdd117bc8503: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:04.763317 23152 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54f560782b1848f698f3bfe3d63c490c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b75f7c127143dca5a81f6708400c37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40257 } } peers { permanent_uuid: "54f560782b1848f698f3bfe3d63c490c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46705 } } peers { permanent_uuid: "8c1c16e258ad4cf885edcdd117bc8503" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40553 } }
I20260619 16:46:04.764022 23227 log.cc:824] T 00000000000000000000000000000000 P 8c1c16e258ad4cf885edcdd117bc8503: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:04.765818 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:46:04.766371 23227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c1c16e258ad4cf885edcdd117bc8503: No bootstrap required, opened a new log
I20260619 16:46:04.767237 23051 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "98b75f7c127143dca5a81f6708400c37"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48254
I20260619 16:46:04.767328 23152 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54f560782b1848f698f3bfe3d63c490c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:04.767477 23051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98b75f7c127143dca5a81f6708400c37 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 16:46:04.767580 23051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98b75f7c127143dca5a81f6708400c37 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:04.769076 23051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98b75f7c127143dca5a81f6708400c37 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b75f7c127143dca5a81f6708400c37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40257 } } peers { permanent_uuid: "54f560782b1848f698f3bfe3d63c490c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46705 } } peers { permanent_uuid: "8c1c16e258ad4cf885edcdd117bc8503" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40553 } }
I20260619 16:46:04.769958 23249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54f560782b1848f698f3bfe3d63c490c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b75f7c127143dca5a81f6708400c37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40257 } } peers { permanent_uuid: "54f560782b1848f698f3bfe3d63c490c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46705 } } peers { permanent_uuid: "8c1c16e258ad4cf885edcdd117bc8503" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40553 } } }
I20260619 16:46:04.770128 23249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54f560782b1848f698f3bfe3d63c490c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:46:04.770812 23126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98b75f7c127143dca5a81f6708400c37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "54f560782b1848f698f3bfe3d63c490c"
I20260619 16:46:04.771070 23126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54f560782b1848f698f3bfe3d63c490c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:04.771780 23227 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c1c16e258ad4cf885edcdd117bc8503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b75f7c127143dca5a81f6708400c37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40257 } } peers { permanent_uuid: "54f560782b1848f698f3bfe3d63c490c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46705 } } peers { permanent_uuid: "8c1c16e258ad4cf885edcdd117bc8503" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40553 } }
I20260619 16:46:04.772112 23227 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c1c16e258ad4cf885edcdd117bc8503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1c16e258ad4cf885edcdd117bc8503, State: Initialized, Role: FOLLOWER
I20260619 16:46:04.772519 23227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c1c16e258ad4cf885edcdd117bc8503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b75f7c127143dca5a81f6708400c37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40257 } } peers { permanent_uuid: "54f560782b1848f698f3bfe3d63c490c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46705 } } peers { permanent_uuid: "8c1c16e258ad4cf885edcdd117bc8503" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40553 } }
I20260619 16:46:04.772652 23126 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 54f560782b1848f698f3bfe3d63c490c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98b75f7c127143dca5a81f6708400c37 in term 1.
I20260619 16:46:04.773706 23027 leader_election.cc:302] T 00000000000000000000000000000000 P 98b75f7c127143dca5a81f6708400c37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54f560782b1848f698f3bfe3d63c490c, 98b75f7c127143dca5a81f6708400c37; no voters:
I20260619 16:46:04.774053 23235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98b75f7c127143dca5a81f6708400c37 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:04.774158 23227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c1c16e258ad4cf885edcdd117bc8503 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:04.774294 23235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98b75f7c127143dca5a81f6708400c37 [term 1 LEADER]: Becoming Leader. State: Replica: 98b75f7c127143dca5a81f6708400c37, State: Running, Role: LEADER
I20260619 16:46:04.774679 23235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98b75f7c127143dca5a81f6708400c37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b75f7c127143dca5a81f6708400c37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40257 } } peers { permanent_uuid: "54f560782b1848f698f3bfe3d63c490c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46705 } } peers { permanent_uuid: "8c1c16e258ad4cf885edcdd117bc8503" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40553 } }
I20260619 16:46:04.776954 23235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98b75f7c127143dca5a81f6708400c37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98b75f7c127143dca5a81f6708400c37. Latest consensus state: current_term: 1 leader_uuid: "98b75f7c127143dca5a81f6708400c37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b75f7c127143dca5a81f6708400c37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40257 } } peers { permanent_uuid: "54f560782b1848f698f3bfe3d63c490c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46705 } } peers { permanent_uuid: "8c1c16e258ad4cf885edcdd117bc8503" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40553 } } }
I20260619 16:46:04.777120 23235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98b75f7c127143dca5a81f6708400c37 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:04.777523 23276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:46:04.778151 23200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98b75f7c127143dca5a81f6708400c37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8c1c16e258ad4cf885edcdd117bc8503"
I20260619 16:46:04.778429 23200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c1c16e258ad4cf885edcdd117bc8503 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:04.778708 23276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:46:04.780076 23200 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8c1c16e258ad4cf885edcdd117bc8503 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98b75f7c127143dca5a81f6708400c37 in term 1.
I20260619 16:46:04.777520 23051 leader_election.cc:288] T 00000000000000000000000000000000 P 98b75f7c127143dca5a81f6708400c37 [CANDIDATE]: Term 1 election: Requested vote from peers 54f560782b1848f698f3bfe3d63c490c (127.0.98.61:46705), 8c1c16e258ad4cf885edcdd117bc8503 (127.0.98.60:40553)
I20260619 16:46:04.776695 23084 master_runner.cc:186] Error getting master registration for 127.0.98.60:40553: OK, instance_id { permanent_uuid: "8c1c16e258ad4cf885edcdd117bc8503" instance_seqno: 1781887564699747 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:46:04.780891 23084 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 16:46:04.786331 23261 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 54f560782b1848f698f3bfe3d63c490c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:46:04.786501 23261 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:46:04.788823 23200 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8c1c16e258ad4cf885edcdd117bc8503 [term 1 FOLLOWER]: Refusing update from remote peer 98b75f7c127143dca5a81f6708400c37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:04.788822 23126 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 54f560782b1848f698f3bfe3d63c490c [term 1 FOLLOWER]: Refusing update from remote peer 98b75f7c127143dca5a81f6708400c37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:04.789737 23263 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 98b75f7c127143dca5a81f6708400c37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c1c16e258ad4cf885edcdd117bc8503" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:04.790320 23235 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 98b75f7c127143dca5a81f6708400c37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f560782b1848f698f3bfe3d63c490c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:04.796679 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.783759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52670 (local address 127.0.98.61:46705)
0619 16:46:04.783883 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:04.783888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:04.785406 (+ 1518us) server_negotiation.cc:407] Connection header received
0619 16:46:04.785609 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:04.785612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:04.785663 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:04.785737 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:04.787118 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.787907 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.790043 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.790370 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.790451 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:04.792568 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:04.792572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:04.792575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:04.792606 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:04.794576 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.795132 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.795135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.795137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.795223 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.795534 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.795537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.795539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.795827 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:04.795931 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:04.796322 (+ 391us) server_negotiation.cc:299] Negotiation successful
0619 16:46:04.796471 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:46:04.797636 23249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54f560782b1848f698f3bfe3d63c490c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98b75f7c127143dca5a81f6708400c37. Latest consensus state: current_term: 1 leader_uuid: "98b75f7c127143dca5a81f6708400c37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b75f7c127143dca5a81f6708400c37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40257 } } peers { permanent_uuid: "54f560782b1848f698f3bfe3d63c490c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46705 } } peers { permanent_uuid: "8c1c16e258ad4cf885edcdd117bc8503" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40553 } } }
I20260619 16:46:04.797837 23249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54f560782b1848f698f3bfe3d63c490c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:46:04.804299 23249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54f560782b1848f698f3bfe3d63c490c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "98b75f7c127143dca5a81f6708400c37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b75f7c127143dca5a81f6708400c37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40257 } } peers { permanent_uuid: "54f560782b1848f698f3bfe3d63c490c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46705 } } peers { permanent_uuid: "8c1c16e258ad4cf885edcdd117bc8503" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40553 } } }
I20260619 16:46:04.804484 23249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54f560782b1848f698f3bfe3d63c490c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:46:04.804912 23262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c1c16e258ad4cf885edcdd117bc8503 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98b75f7c127143dca5a81f6708400c37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b75f7c127143dca5a81f6708400c37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40257 } } peers { permanent_uuid: "54f560782b1848f698f3bfe3d63c490c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46705 } } peers { permanent_uuid: "8c1c16e258ad4cf885edcdd117bc8503" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40553 } } }
I20260619 16:46:04.805274 23262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c1c16e258ad4cf885edcdd117bc8503 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:46:04.805434 23262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c1c16e258ad4cf885edcdd117bc8503 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98b75f7c127143dca5a81f6708400c37. Latest consensus state: current_term: 1 leader_uuid: "98b75f7c127143dca5a81f6708400c37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b75f7c127143dca5a81f6708400c37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40257 } } peers { permanent_uuid: "54f560782b1848f698f3bfe3d63c490c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46705 } } peers { permanent_uuid: "8c1c16e258ad4cf885edcdd117bc8503" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40553 } } }
I20260619 16:46:04.805877 23262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c1c16e258ad4cf885edcdd117bc8503 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:46:04.805884 23235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98b75f7c127143dca5a81f6708400c37 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "98b75f7c127143dca5a81f6708400c37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b75f7c127143dca5a81f6708400c37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40257 } } peers { permanent_uuid: "54f560782b1848f698f3bfe3d63c490c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46705 } } peers { permanent_uuid: "8c1c16e258ad4cf885edcdd117bc8503" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40553 } } }
I20260619 16:46:04.806034 23235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98b75f7c127143dca5a81f6708400c37 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:04.807792 23288 mvcc.cc:205] Tried to move back new op lower bound from 7298611465350422528 to 7298611465321529344. Current Snapshot: MvccSnapshot[applied={T|T < 7298611465350422528}]
W20260619 16:46:04.808118 23275 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8c1c16e258ad4cf885edcdd117bc8503: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:46:04.808362 23275 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:46:04.817883 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.790146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55404 (local address 127.0.98.60:40553)
0619 16:46:04.790273 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:04.790279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:04.806886 (+ 16607us) server_negotiation.cc:407] Connection header received
0619 16:46:04.806938 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:04.806942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:04.806996 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:04.807071 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:04.808149 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.809178 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.810437 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.810797 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.810895 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:04.812943 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:04.812948 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:04.812951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:04.812985 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:04.815641 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.816186 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.816190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.816192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.816284 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.816561 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.816564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.816566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.816804 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:04.816944 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:04.817576 (+ 632us) server_negotiation.cc:299] Negotiation successful
0619 16:46:04.817714 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:46:04.822774 23276 catalog_manager.cc:1347] Generated new cluster ID: efa0d5b7481b4fd48e4e30071aeb2080
I20260619 16:46:04.822866 23276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:46:04.828397 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.784844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48256 (local address 127.0.98.62:40257)
0619 16:46:04.811711 (+ 26867us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:04.811717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:04.811734 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:46:04.811782 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:04.811785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:04.813320 (+ 1535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:04.814602 (+ 1282us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:04.814771 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.820110 (+ 5339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.821379 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.821785 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.822350 (+ 565us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:04.823830 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:04.823834 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:04.823837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:04.823868 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:04.825617 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.826286 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.826289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.826291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.826388 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.826730 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.826733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.826735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.826974 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:04.827912 (+ 938us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:04.827995 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 16:46:04.828214 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26685}
I20260619 16:46:04.836817 23285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98b75f7c127143dca5a81f6708400c37 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "98b75f7c127143dca5a81f6708400c37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b75f7c127143dca5a81f6708400c37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40257 } } peers { permanent_uuid: "54f560782b1848f698f3bfe3d63c490c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46705 } } peers { permanent_uuid: "8c1c16e258ad4cf885edcdd117bc8503" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40553 } } }
I20260619 16:46:04.837011 23285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98b75f7c127143dca5a81f6708400c37 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:04.838735 23287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c1c16e258ad4cf885edcdd117bc8503 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "98b75f7c127143dca5a81f6708400c37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b75f7c127143dca5a81f6708400c37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40257 } } peers { permanent_uuid: "54f560782b1848f698f3bfe3d63c490c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46705 } } peers { permanent_uuid: "8c1c16e258ad4cf885edcdd117bc8503" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40553 } } }
I20260619 16:46:04.839547 23287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c1c16e258ad4cf885edcdd117bc8503 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:46:04.839805 23276 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:46:04.841605 23276 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:46:04.842461 23290 mvcc.cc:205] Tried to move back new op lower bound from 7298611465350422528 to 7298611465321529344. Current Snapshot: MvccSnapshot[applied={T|T < 7298611465350422528}]
I20260619 16:46:04.856221 23276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98b75f7c127143dca5a81f6708400c37: Generated new TSK 0
I20260619 16:46:04.858402 23276 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:46:04.896956 23230 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:04.892864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55416 (local address 127.0.98.60:40553)
0619 16:46:04.893006 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:04.893011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:04.895461 (+ 2450us) server_negotiation.cc:407] Connection header received
0619 16:46:04.895700 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:04.895704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:04.895919 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:04.895997 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:04.896790 (+ 793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55416: BlockingRecv error: recv got EOF from 127.0.0.1:55416 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 16:46:04.899634 23233 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:04.894822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48260 (local address 127.0.98.62:40257)
0619 16:46:04.894939 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:04.894944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:04.897997 (+ 3053us) server_negotiation.cc:407] Connection header received
0619 16:46:04.898285 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:04.898290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:04.898346 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:04.898401 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:04.899492 (+ 1091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48260: BlockingRecv error: recv got EOF from 127.0.0.1:48260 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 16:46:04.900700 23156 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:04.893858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52684 (local address 127.0.98.61:46705)
0619 16:46:04.893989 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:04.893994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:04.897814 (+ 3820us) server_negotiation.cc:407] Connection header received
0619 16:46:04.898777 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:04.898781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:04.898840 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:04.898896 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:04.900572 (+ 1676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52684: BlockingRecv error: recv got EOF from 127.0.0.1:52684 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 16:46:04.906204 23156 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:04.904374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52694 (local address 127.0.98.61:46705)
0619 16:46:04.904493 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:04.904498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:04.905033 (+ 535us) server_negotiation.cc:407] Connection header received
0619 16:46:04.905312 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:04.905316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:04.905399 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:04.905453 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:04.906071 (+ 618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52694: BlockingRecv error: recv got EOF from 127.0.0.1:52694 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 16:46:04.911877 23156 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:04.910072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52702 (local address 127.0.98.61:46705)
0619 16:46:04.910207 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:04.910212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:04.910800 (+ 588us) server_negotiation.cc:407] Connection header received
0619 16:46:04.911017 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:04.911021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:04.911079 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:04.911135 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:04.911752 (+ 617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52702: BlockingRecv error: recv got EOF from 127.0.0.1:52702 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 16:46:04.918488 23156 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:04.916547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52704 (local address 127.0.98.61:46705)
0619 16:46:04.916677 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:04.916681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:04.917378 (+ 697us) server_negotiation.cc:407] Connection header received
0619 16:46:04.917609 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:04.917613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:04.917671 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:04.917740 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:04.918366 (+ 626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52704: BlockingRecv error: recv got EOF from 127.0.0.1:52704 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 16:46:04.927719 23156 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:04.925697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52708 (local address 127.0.98.61:46705)
0619 16:46:04.925873 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:04.925878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:04.926604 (+ 726us) server_negotiation.cc:407] Connection header received
0619 16:46:04.926833 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:04.926836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:04.926892 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:04.926950 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:04.927563 (+ 613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52708: BlockingRecv error: recv got EOF from 127.0.0.1:52708 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260619 16:46:04.940675 23156 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:04.938618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52710 (local address 127.0.98.61:46705)
0619 16:46:04.938768 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:04.938774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:04.939423 (+ 649us) server_negotiation.cc:407] Connection header received
0619 16:46:04.939741 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:04.939744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:04.939808 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:04.939868 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:04.940543 (+ 675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52710: BlockingRecv error: recv got EOF from 127.0.0.1:52710 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 16:46:04.961792 23156 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:04.959745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52716 (local address 127.0.98.61:46705)
0619 16:46:04.959891 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:04.959896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:04.960769 (+ 873us) server_negotiation.cc:407] Connection header received
0619 16:46:04.960982 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:04.960986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:04.961043 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:04.961096 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:04.961655 (+ 559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52716: BlockingRecv error: recv got EOF from 127.0.0.1:52716 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 16:46:04.997937 23156 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:04.996572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52726 (local address 127.0.98.61:46705)
0619 16:46:04.996707 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:04.996711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:04.997004 (+ 293us) server_negotiation.cc:407] Connection header received
0619 16:46:04.997176 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:04.997179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:04.997235 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:04.997335 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:04.997808 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52726: BlockingRecv error: recv got EOF from 127.0.0.1:52726 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 16:46:05.065559 23156 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:05.064357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52742 (local address 127.0.98.61:46705)
0619 16:46:05.064543 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:05.064546 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:05.064641 (+ 95us) server_negotiation.cc:407] Connection header received
0619 16:46:05.064741 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:05.064744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:05.064797 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:05.064907 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:05.065455 (+ 548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52742: BlockingRecv error: recv got EOF from 127.0.0.1:52742 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260619 16:46:05.197423 23156 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:05.196100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52748 (local address 127.0.98.61:46705)
0619 16:46:05.196281 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:05.196287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:05.196482 (+ 195us) server_negotiation.cc:407] Connection header received
0619 16:46:05.196661 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:05.196665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:05.196728 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:05.196825 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:05.197311 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52748: BlockingRecv error: recv got EOF from 127.0.0.1:52748 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260619 16:46:05.457617 23156 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:05.456255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52758 (local address 127.0.98.61:46705)
0619 16:46:05.456431 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:05.456435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:05.456627 (+ 192us) server_negotiation.cc:407] Connection header received
0619 16:46:05.456724 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:05.456727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:05.456790 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:05.456976 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:05.457433 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52758: BlockingRecv error: recv got EOF from 127.0.0.1:52758 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:46:05.788373 23261 catalog_manager.cc:1259] Loaded cluster ID: efa0d5b7481b4fd48e4e30071aeb2080
I20260619 16:46:05.788499 23261 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 54f560782b1848f698f3bfe3d63c490c: loading cluster ID for follower catalog manager: success
I20260619 16:46:05.790567 23261 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 54f560782b1848f698f3bfe3d63c490c: acquiring CA information for follower catalog manager: success
I20260619 16:46:05.792372 23261 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 54f560782b1848f698f3bfe3d63c490c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:46:05.810195 23275 catalog_manager.cc:1259] Loaded cluster ID: efa0d5b7481b4fd48e4e30071aeb2080
I20260619 16:46:05.810302 23275 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8c1c16e258ad4cf885edcdd117bc8503: loading cluster ID for follower catalog manager: success
I20260619 16:46:05.812865 23275 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8c1c16e258ad4cf885edcdd117bc8503: acquiring CA information for follower catalog manager: success
I20260619 16:46:05.814503 23275 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8c1c16e258ad4cf885edcdd117bc8503: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:46:05.977969 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:05.972522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52764 (local address 127.0.98.61:46705)
0619 16:46:05.972858 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:05.972864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:05.973045 (+ 181us) server_negotiation.cc:407] Connection header received
0619 16:46:05.973261 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:05.973265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:05.973444 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:05.973587 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:05.973872 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:05.974560 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:05.975835 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:05.976138 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:05.976504 (+ 366us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:05.976877 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:05.977417 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:05.977527 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:05.977666 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 16:46:05.977738 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":100,"threads_started":1}
I20260619 16:46:05.987432 23368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:05.982359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55418 (local address 127.0.98.60:40553)
0619 16:46:05.982633 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:05.982636 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:05.982653 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:46:05.982858 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:05.982861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:05.982999 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:05.983117 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:05.983406 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:05.984029 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:05.985101 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:05.985653 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:05.985806 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:05.986270 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:05.986951 (+ 681us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:05.987108 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:05.987145 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 16:46:05.987227 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":74,"threads_started":1}
I20260619 16:46:05.996408 23368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:05.991478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55430 (local address 127.0.98.60:40553)
0619 16:46:05.991700 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:05.991704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:05.992002 (+ 298us) server_negotiation.cc:407] Connection header received
0619 16:46:05.992258 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:05.992261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:05.992318 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:05.992418 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:05.993006 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:05.993927 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:05.995045 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:05.995333 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:05.995442 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:05.995863 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:05.995964 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:05.996014 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:05.996168 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 16:46:05.996233 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:46:05.996757 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:05.991297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52774 (local address 127.0.98.61:46705)
0619 16:46:05.991508 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:05.991515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:05.992018 (+ 503us) server_negotiation.cc:407] Connection header received
0619 16:46:05.992587 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:05.992590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:05.992697 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:05.992780 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:05.993198 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:05.994012 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:05.995306 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:05.995623 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:05.995764 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:05.996154 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:05.996267 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:05.996342 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:05.996530 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 16:46:05.996607 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260619 16:46:05.997239 23374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:05.991209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48268 (local address 127.0.98.62:40257)
0619 16:46:05.991514 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:05.991520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:05.991968 (+ 448us) server_negotiation.cc:407] Connection header received
0619 16:46:05.992210 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:05.992216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:05.992436 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:05.992562 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:05.993102 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:05.993855 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:05.995333 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:05.995597 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:05.995775 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:05.996162 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:05.996639 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:05.996732 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:05.996933 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 16:46:05.997014 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":91,"threads_started":1}
I20260619 16:46:06.007731 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 23009
I20260619 16:46:06.015146 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 23084
I20260619 16:46:06.021080 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8rFIPy/build/debug/bin/kudu with pid 23158
2026-06-19T16:46:06Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3215 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6587 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (306434 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST